Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11251

ConcurrentModificationException during DataNode#refreshVolumes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None

      Description

      The testAddVolumesDuringWrite case failed with a ReconfigurationException which appears to have been caused by a ConcurrentModificationException. Stacktrace details to follow.

      1. HDFS-11251.01.patch
        5 kB
        Manoj Govindassamy
      2. HDFS-11251.02.patch
        12 kB
        Manoj Govindassamy
      3. HDFS-11251.03.patch
        13 kB
        Manoj Govindassamy
      4. HDFS-11251.04.patch
        13 kB
        Manoj Govindassamy

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11054 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11054/)
        HDFS-11251. ConcurrentModificationException during (lei: rev e9f1396834174646a8d7aa8fc6c4a4f724ca5b28)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11054 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11054/ ) HDFS-11251 . ConcurrentModificationException during (lei: rev e9f1396834174646a8d7aa8fc6c4a4f724ca5b28) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        +1. Thanks Manoj Govindassamy

        The last patch passed test here and the jenkins failure is not related.

        Committed to trunk and branch-2.

        Show
        eddyxu Lei (Eddy) Xu added a comment - +1. Thanks Manoj Govindassamy The last patch passed test here and the jenkins failure is not related. Committed to trunk and branch-2 .
        Hide
        manojg Manoj Govindassamy added a comment -

        Fixed the DEFAULT_STORAGES_PER_DATANODE to 2 in TestDataNodeHotSwapVolumes. Uploaded the patch v04.
        Lei (Eddy) Xu, can you please take a look ?

        Show
        manojg Manoj Govindassamy added a comment - Fixed the DEFAULT_STORAGES_PER_DATANODE to 2 in TestDataNodeHotSwapVolumes. Uploaded the patch v04. Lei (Eddy) Xu , can you please take a look ?
        Hide
        manojg Manoj Govindassamy added a comment -

        Thanks for verifying the fix. The latest patch has one unintentional change on the constant. Fixing it.

        Show
        manojg Manoj Govindassamy added a comment - Thanks for verifying the fix. The latest patch has one unintentional change on the constant. Fixing it.
        Hide
        manojg Manoj Govindassamy added a comment -

        Oops, not intentional. Copy paste from the previous line mistake. 2 is the desired number here. Thanks for catching this. Will upload the new patch soon.

        Show
        manojg Manoj Govindassamy added a comment - Oops, not intentional. Copy paste from the previous line mistake. 2 is the desired number here. Thanks for catching this. Will upload the new patch soon.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 12m 34s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 150 unchanged - 2 fixed = 151 total (was 152)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 1s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 163m 8s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        187m 17s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11251
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844856/HDFS-11251.03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ea28270ca463 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9262797
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17965/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17965/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17965/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 34s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 150 unchanged - 2 fixed = 151 total (was 152) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 163m 8s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 187m 17s Reason Tests Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11251 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844856/HDFS-11251.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ea28270ca463 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9262797 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17965/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17965/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17965/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17965/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        It seems this change lead this

        +  private static final int DEFAULT_STORAGES_PER_DATANODE = 512;
           private MiniDFSCluster cluster;
        

        Is that your intended change here, Manoj Govindassamy? Actually the default value 2 is enough here since startDFSCluster(int numNameNodes, int numDataNodes) is used by other tests.

        Show
        linyiqun Yiqun Lin added a comment - It seems this change lead this + private static final int DEFAULT_STORAGES_PER_DATANODE = 512; private MiniDFSCluster cluster; Is that your intended change here, Manoj Govindassamy ? Actually the default value 2 is enough here since startDFSCluster(int numNameNodes, int numDataNodes) is used by other tests.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Hi, Manoj Govindassamy

        The latest patch hangs on my local machine:

        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
        
        
        
        
        ====> TEST TIMED OUT. PRINTING THREAD DUMP. <====
        Timestamp: 2016-12-28 01:41:15,000
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data34)" daemon prio=5 tid=34502 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "pool-3452-thread-1"  prio=5 tid=30550 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server Responder" daemon prio=5 tid=28901 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1328)
                at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1311)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data58)" daemon prio=5 tid=34527 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 8 on 50718" daemon prio=5 tid=30559 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data14)" daemon prio=5 tid=34482 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data86)" daemon prio=5 tid=34556 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "qtp282733206-32102" daemon prio=5 tid=32102 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
                at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        "org.eclipse.jetty.server.session.HashSessionManager@3bfa7babTimer" daemon prio=5 tid=30538 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data20)" daemon prio=5 tid=34488 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 7 on 50708" daemon prio=5 tid=28914 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "qtp1387697241-30532-acceptor-0@1d1c322a-ServerConnector@6ca16f0f{HTTP/1.1,[http/1.1]}{localhost:50716}" daemon prio=3 tid=30532 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
                at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
                at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
                at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)
                at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data78)" daemon prio=5 tid=34548 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 7 on 50723" daemon prio=5 tid=32129 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data52)" daemon prio=5 tid=34521 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data26)" daemon prio=5 tid=34494 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data17)" daemon prio=5 tid=34485 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data83)" daemon prio=5 tid=34553 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data76)" daemon prio=5 tid=34546 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 9 on 50718" daemon prio=5 tid=30560 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "org.eclipse.jetty.server.session.HashSessionManager@4fab38daTimer" daemon prio=5 tid=30540 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server listener on 50723" daemon prio=5 tid=32114 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1150)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data23)" daemon prio=5 tid=34491 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "qtp1387697241-30535" daemon prio=5 tid=30535 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data70)" daemon prio=5 tid=34540 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "StorageInfoMonitor" daemon prio=5 tid=28894 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$StorageInfoDefragmenter.run(BlockManager.java:4322)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data65)" daemon prio=5 tid=34534 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data71)" daemon prio=5 tid=34541 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data40)" daemon prio=5 tid=34509 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "Socket Reader #1 for port 50723"  prio=5 tid=32115 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1088)
                at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1067)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data16)" daemon prio=5 tid=34484 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "qtp490528317-28885" daemon prio=5 tid=28885 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data85)" daemon prio=5 tid=34555 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "qtp282733206-32106" daemon prio=5 tid=32106 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "org.eclipse.jetty.server.session.HashSessionManager@4726bdbeTimer" daemon prio=5 tid=32110 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data11)" daemon prio=5 tid=34479 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 1 on 50723" daemon prio=5 tid=32123 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data73)" daemon prio=5 tid=34543 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data59)" daemon prio=5 tid=34528 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data41)" daemon prio=5 tid=34510 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 0 on 50718" daemon prio=5 tid=30551 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "Block report processor" daemon prio=5 tid=28895 in Object.wait()
        java.lang.Thread.State: WAITING (on object monitor)
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
                at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
                at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:4623)
                at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:4612)
        "IPC Server handler 2 on 50718" daemon prio=5 tid=30553 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "qtp282733206-32105" daemon prio=5 tid=32105 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data30)" daemon prio=5 tid=34498 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data8)" daemon prio=5 tid=34476 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "AsyncAppender-Dispatcher-Thread-1083" daemon prio=5 tid=1719 in Object.wait()
        java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:502)
                at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:548)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server handler 0 on 50723" daemon prio=5 tid=32122 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data75)" daemon prio=5 tid=34545 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data50)" daemon prio=5 tid=34519 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "qtp1387697241-30536" daemon prio=5 tid=30536 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "qtp490528317-28883" daemon prio=5 tid=28883 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
                at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        "qtp282733206-32108" daemon prio=5 tid=32108 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data19)" daemon prio=5 tid=34487 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data57)" daemon prio=5 tid=34526 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data81)" daemon prio=5 tid=34551 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data33)" daemon prio=5 tid=34501 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server idle connection scanner for port 50708" daemon prio=5 tid=28900 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at java.util.TimerThread.mainLoop(Timer.java:552)
                at java.util.TimerThread.run(Timer.java:505)
        "qtp490528317-28882" daemon prio=5 tid=28882 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
                at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data21)" daemon prio=5 tid=34489 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "qtp490528317-28889" daemon prio=5 tid=28889 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2)" daemon prio=5 tid=34470 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data74)" daemon prio=5 tid=34544 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data69)" daemon prio=5 tid=34538 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data68)" daemon prio=5 tid=34537 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data48)" daemon prio=5 tid=34517 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@629b7761" daemon prio=5 tid=30528 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
                at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
                at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
                at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100)
                at org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:85)
                at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:145)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data84)" daemon prio=5 tid=34554 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.apache.hadoop.util.JvmPauseMonitor$Monitor@78a9fb45" daemon prio=5 tid=30542 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server handler 7 on 50718" daemon prio=5 tid=30558 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "IPC Server Responder" daemon prio=5 tid=30546 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1328)
                at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1311)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data13)" daemon prio=5 tid=34481 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@22760570" daemon prio=5 tid=28919 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:339)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server handler 5 on 50718" daemon prio=5 tid=30556 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data77)" daemon prio=5 tid=34547 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data6)" daemon prio=5 tid=34474 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 1 on 50718" daemon prio=5 tid=30552 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "pool-3451-thread-1"  prio=5 tid=30529 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "org.apache.hadoop.util.JvmPauseMonitor$Monitor@7ab86d15" daemon prio=5 tid=32113 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data38)" daemon prio=5 tid=34507 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data29)" daemon prio=5 tid=34497 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.eclipse.jetty.server.session.HashSessionManager@604680beTimer" daemon prio=5 tid=28891 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "qtp282733206-32101" daemon prio=5 tid=32101 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
                at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data79)" daemon prio=5 tid=34549 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7)" daemon prio=5 tid=34475 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "Socket Reader #1 for port 50708"  prio=5 tid=28899 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1088)
                at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1067)
        "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@7029c709" daemon prio=5 tid=28920 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3752)
                at java.lang.Thread.run(Thread.java:745)
        "RedundancyMonitor" daemon prio=5 tid=28893 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at java.lang.Thread.sleep(Thread.java:340)
                at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
                at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$RedundancyMonitor.run(BlockManager.java:4287)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data72)" daemon prio=5 tid=34542 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "main"  prio=5 tid=1 runnable
        java.lang.Thread.State: RUNNABLE
                at java.lang.Thread.dumpThreads(Native Method)
                at java.lang.Thread.getAllStackTraces(Thread.java:1607)
                at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87)
                at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73)
                at org.apache.hadoop.test.TimedOutTestsListener.testFailure(TimedOutTestsListener.java:62)
                at org.junit.runner.notification.RunNotifier$4.notifyListener(RunNotifier.java:139)
                at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:61)
                at org.junit.runner.notification.RunNotifier.fireTestFailures(RunNotifier.java:134)
                at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:128)
                at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:23)
                at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:275)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
                at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
                at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
                at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
                at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
                at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
                at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
                at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
                at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
                at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
                at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
                at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
                at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
                at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
        "Reference Handler" daemon prio=10 tid=2 in Object.wait()
        java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:502)
                at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
                at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
        "qtp1387697241-30531" daemon prio=5 tid=30531 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
                at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server handler 8 on 50708" daemon prio=5 tid=28915 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "IPC Server idle connection scanner for port 50723" daemon prio=5 tid=32116 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at java.util.TimerThread.mainLoop(Timer.java:552)
                at java.util.TimerThread.run(Timer.java:505)
        "pool-3448-thread-1"  prio=5 tid=28917 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data37)" daemon prio=5 tid=34506 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data43)" daemon prio=5 tid=34512 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 2 on 50708" daemon prio=5 tid=28909 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "Timer for 'NameNode' metrics system" daemon prio=5 tid=28879 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at java.util.TimerThread.mainLoop(Timer.java:552)
                at java.util.TimerThread.run(Timer.java:505)
        "IPC Server Responder" daemon prio=5 tid=32117 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1328)
                at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1311)
        "qtp282733206-32107" daemon prio=5 tid=32107 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "qtp1387697241-30537" daemon prio=5 tid=30537 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "pool-3446-thread-1"  prio=5 tid=28881 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server handler 5 on 50708" daemon prio=5 tid=28912 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data28)" daemon prio=5 tid=34496 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@6ffc0448" daemon prio=5 tid=28896 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:451)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data45)" daemon prio=5 tid=34514 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1)" daemon prio=5 tid=34468 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data55)" daemon prio=5 tid=34524 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data62)" daemon prio=5 tid=34531 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 1 on 50708" daemon prio=5 tid=28908 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data60)" daemon prio=5 tid=34529 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data42)" daemon prio=5 tid=34511 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "Attach Listener" daemon prio=9 tid=27524 runnable
        java.lang.Thread.State: RUNNABLE
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data87)" daemon prio=5 tid=34557 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data10)" daemon prio=5 tid=34478 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data67)" daemon prio=5 tid=34536 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data61)" daemon prio=5 tid=34530 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 6 on 50708" daemon prio=5 tid=28913 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data63)" daemon prio=5 tid=34532 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@7ef2b559" daemon prio=5 tid=28922 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:3881)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data44)" daemon prio=5 tid=34513 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "pool-3457-thread-1"  prio=5 tid=32100 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "org.eclipse.jetty.server.session.HashSessionManager@6a64387cTimer" daemon prio=5 tid=32109 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "qtp490528317-28888" daemon prio=5 tid=28888 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "qtp490528317-28887" daemon prio=5 tid=28887 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server handler 3 on 50723" daemon prio=5 tid=32125 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "CacheReplicationMonitor(1664877657)"  prio=5 tid=28923 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
                at org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data4)" daemon prio=5 tid=34472 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.eclipse.jetty.server.session.HashSessionManager@5bb89342Timer" daemon prio=5 tid=28892 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data36)" daemon prio=5 tid=34505 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 0 on 50708" daemon prio=5 tid=28907 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data12)" daemon prio=5 tid=34480 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data15)" daemon prio=5 tid=34483 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3)" daemon prio=5 tid=34471 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data64)" daemon prio=5 tid=34533 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data18)" daemon prio=5 tid=34486 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "nioEventLoopGroup-22-1"  prio=10 tid=30541 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:621)
                at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:309)
                at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:703)
                at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
                at java.lang.Thread.run(Thread.java:745)
        "Socket Reader #1 for port 50718"  prio=5 tid=30544 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1088)
                at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1067)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data35)" daemon prio=5 tid=34504 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=5 tid=64 in Object.wait()
        java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
                at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3698)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data47)" daemon prio=5 tid=34516 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.eclipse.jetty.server.session.HashSessionManager@1c38d141Timer" daemon prio=5 tid=32111 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "DecommissionMonitor-0" daemon prio=5 tid=28905 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data9)" daemon prio=5 tid=34477 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 9 on 50723" daemon prio=5 tid=32131 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "org.eclipse.jetty.server.session.HashSessionManager@2eaae4c6Timer" daemon prio=5 tid=28890 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data49)" daemon prio=5 tid=34518 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 2 on 50723" daemon prio=5 tid=32124 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "IPC Server handler 6 on 50723" daemon prio=5 tid=32128 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "IPC Server listener on 50708" daemon prio=5 tid=28898 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1150)
        "Thread-26471"  prio=5 tid=34580 runnable
        java.lang.Thread.State: RUNNABLE
                at org.eclipse.jetty.util.thread.QueuedThreadPool.doStop(QueuedThreadPool.java:181)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
                at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
                at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
                at org.eclipse.jetty.server.Server.doStop(Server.java:482)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
                at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1071)
                at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.stop(NameNodeHttpServer.java:243)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.stopHttpServer(NameNode.java:884)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.stopCommonServices(NameNode.java:841)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1009)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.stopAtException(NameNode.java:960)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:949)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:919)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1636)
                at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1263)
                at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1032)
                at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:907)
                at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:839)
                at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:491)
                at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:450)
                at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.startDFSCluster(TestDataNodeHotSwapVolumes.java:137)
                at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.startDFSCluster(TestDataNodeHotSwapVolumes.java:108)
                at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:421)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
                at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
                at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
                at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
                at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data39)" daemon prio=5 tid=34508 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@6b2cbc8f" daemon prio=5 tid=28921 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:3794)
                at java.lang.Thread.run(Thread.java:745)
        "qtp1387697241-30530" daemon prio=5 tid=30530 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
                at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
                at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
                at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
                at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
                at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
                at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        "IPC Server handler 4 on 50723" daemon prio=5 tid=32126 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data54)" daemon prio=5 tid=34523 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data80)" daemon prio=5 tid=34550 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 3 on 50708" daemon prio=5 tid=28910 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "org.apache.hadoop.util.JvmPauseMonitor$Monitor@7e0a2aa" daemon prio=5 tid=28880 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Thread.sleep(Native Method)
                at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192)
                at java.lang.Thread.run(Thread.java:745)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data56)" daemon prio=5 tid=34525 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "IPC Server handler 5 on 50723" daemon prio=5 tid=32127 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at sun.misc.Unsafe.park(Native Method)
                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
                at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
                at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639)
        "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data82)" daemon prio=5 tid=34552 timed_waiting
        java.lang.Thread.State: TIMED_WAITING
                at java.lang.Object.wait(Native Method)
                at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620)
        "qtp282733206-32103-acceptor-0@59ab2598-ServerConnector@2635a54a{HTTP/1.1,[http/1.1]}{localhost:50721}" daemon prio=3 tid=32103 runnable
        java.lang.Thread.State: RUNNABLE
                at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
                at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
                at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
                at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)
                at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
                at java.lang.Thread.run(Thread.java:745)
        
        Show
        eddyxu Lei (Eddy) Xu added a comment - Hi, Manoj Govindassamy The latest patch hangs on my local machine: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes ====> TEST TIMED OUT. PRINTING THREAD DUMP. <==== Timestamp: 2016-12-28 01:41:15,000 "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data34)" daemon prio=5 tid=34502 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "pool-3452-thread-1" prio=5 tid=30550 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "IPC Server Responder" daemon prio=5 tid=28901 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1328) at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1311) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data58)" daemon prio=5 tid=34527 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 8 on 50718" daemon prio=5 tid=30559 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data14)" daemon prio=5 tid=34482 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data86)" daemon prio=5 tid=34556 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "qtp282733206-32102" daemon prio=5 tid=32102 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745) "org.eclipse.jetty.server.session.HashSessionManager@3bfa7babTimer" daemon prio=5 tid=30538 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data20)" daemon prio=5 tid=34488 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 7 on 50708" daemon prio=5 tid=28914 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "qtp1387697241-30532-acceptor-0@1d1c322a-ServerConnector@6ca16f0f{HTTP/1.1,[http/1.1]}{localhost:50716}" daemon prio=3 tid=30532 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data78)" daemon prio=5 tid=34548 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 7 on 50723" daemon prio=5 tid=32129 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data52)" daemon prio=5 tid=34521 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data26)" daemon prio=5 tid=34494 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data17)" daemon prio=5 tid=34485 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data83)" daemon prio=5 tid=34553 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data76)" daemon prio=5 tid=34546 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 9 on 50718" daemon prio=5 tid=30560 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "org.eclipse.jetty.server.session.HashSessionManager@4fab38daTimer" daemon prio=5 tid=30540 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "IPC Server listener on 50723" daemon prio=5 tid=32114 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1150) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data23)" daemon prio=5 tid=34491 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "qtp1387697241-30535" daemon prio=5 tid=30535 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data70)" daemon prio=5 tid=34540 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "StorageInfoMonitor" daemon prio=5 tid=28894 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$StorageInfoDefragmenter.run(BlockManager.java:4322) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data65)" daemon prio=5 tid=34534 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data71)" daemon prio=5 tid=34541 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data40)" daemon prio=5 tid=34509 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "Socket Reader #1 for port 50723" prio=5 tid=32115 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1088) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1067) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data16)" daemon prio=5 tid=34484 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "qtp490528317-28885" daemon prio=5 tid=28885 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data85)" daemon prio=5 tid=34555 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "qtp282733206-32106" daemon prio=5 tid=32106 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "org.eclipse.jetty.server.session.HashSessionManager@4726bdbeTimer" daemon prio=5 tid=32110 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data11)" daemon prio=5 tid=34479 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 1 on 50723" daemon prio=5 tid=32123 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data73)" daemon prio=5 tid=34543 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data59)" daemon prio=5 tid=34528 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data41)" daemon prio=5 tid=34510 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 0 on 50718" daemon prio=5 tid=30551 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "Block report processor" daemon prio=5 tid=28895 in Object .wait() java.lang. Thread .State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:4623) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:4612) "IPC Server handler 2 on 50718" daemon prio=5 tid=30553 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "qtp282733206-32105" daemon prio=5 tid=32105 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data30)" daemon prio=5 tid=34498 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data8)" daemon prio=5 tid=34476 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "AsyncAppender-Dispatcher- Thread -1083" daemon prio=5 tid=1719 in Object .wait() java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) at java.lang. Object .wait( Object .java:502) at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:548) at java.lang. Thread .run( Thread .java:745) "IPC Server handler 0 on 50723" daemon prio=5 tid=32122 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data75)" daemon prio=5 tid=34545 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data50)" daemon prio=5 tid=34519 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "qtp1387697241-30536" daemon prio=5 tid=30536 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "qtp490528317-28883" daemon prio=5 tid=28883 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745) "qtp282733206-32108" daemon prio=5 tid=32108 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data19)" daemon prio=5 tid=34487 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data57)" daemon prio=5 tid=34526 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data81)" daemon prio=5 tid=34551 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data33)" daemon prio=5 tid=34501 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server idle connection scanner for port 50708" daemon prio=5 tid=28900 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "qtp490528317-28882" daemon prio=5 tid=28882 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data21)" daemon prio=5 tid=34489 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "qtp490528317-28889" daemon prio=5 tid=28889 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2)" daemon prio=5 tid=34470 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data74)" daemon prio=5 tid=34544 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data69)" daemon prio=5 tid=34538 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data68)" daemon prio=5 tid=34537 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data48)" daemon prio=5 tid=34517 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@629b7761" daemon prio=5 tid=30528 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) at org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:85) at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:145) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data84)" daemon prio=5 tid=34554 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.apache.hadoop.util.JvmPauseMonitor$Monitor@78a9fb45" daemon prio=5 tid=30542 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192) at java.lang. Thread .run( Thread .java:745) "IPC Server handler 7 on 50718" daemon prio=5 tid=30558 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "IPC Server Responder" daemon prio=5 tid=30546 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1328) at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1311) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data13)" daemon prio=5 tid=34481 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@22760570" daemon prio=5 tid=28919 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:339) at java.lang. Thread .run( Thread .java:745) "IPC Server handler 5 on 50718" daemon prio=5 tid=30556 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data77)" daemon prio=5 tid=34547 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data6)" daemon prio=5 tid=34474 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 1 on 50718" daemon prio=5 tid=30552 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "pool-3451-thread-1" prio=5 tid=30529 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "org.apache.hadoop.util.JvmPauseMonitor$Monitor@7ab86d15" daemon prio=5 tid=32113 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data38)" daemon prio=5 tid=34507 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data29)" daemon prio=5 tid=34497 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.eclipse.jetty.server.session.HashSessionManager@604680beTimer" daemon prio=5 tid=28891 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "qtp282733206-32101" daemon prio=5 tid=32101 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data79)" daemon prio=5 tid=34549 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data7)" daemon prio=5 tid=34475 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "Socket Reader #1 for port 50708" prio=5 tid=28899 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1088) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1067) "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@7029c709" daemon prio=5 tid=28920 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3752) at java.lang. Thread .run( Thread .java:745) "RedundancyMonitor" daemon prio=5 tid=28893 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at java.lang. Thread .sleep( Thread .java:340) at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$RedundancyMonitor.run(BlockManager.java:4287) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data72)" daemon prio=5 tid=34542 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "main" prio=5 tid=1 runnable java.lang. Thread .State: RUNNABLE at java.lang. Thread .dumpThreads(Native Method) at java.lang. Thread .getAllStackTraces( Thread .java:1607) at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87) at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73) at org.apache.hadoop.test.TimedOutTestsListener.testFailure(TimedOutTestsListener.java:62) at org.junit.runner.notification.RunNotifier$4.notifyListener(RunNotifier.java:139) at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:61) at org.junit.runner.notification.RunNotifier.fireTestFailures(RunNotifier.java:134) at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:128) at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:23) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:275) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) "Reference Handler" daemon prio=10 tid=2 in Object .wait() java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) at java.lang. Object .wait( Object .java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "qtp1387697241-30531" daemon prio=5 tid=30531 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745) "IPC Server handler 8 on 50708" daemon prio=5 tid=28915 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "IPC Server idle connection scanner for port 50723" daemon prio=5 tid=32116 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "pool-3448-thread-1" prio=5 tid=28917 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data37)" daemon prio=5 tid=34506 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data43)" daemon prio=5 tid=34512 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 2 on 50708" daemon prio=5 tid=28909 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "Timer for 'NameNode' metrics system" daemon prio=5 tid=28879 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "IPC Server Responder" daemon prio=5 tid=32117 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:1328) at org.apache.hadoop.ipc.Server$Responder.run(Server.java:1311) "qtp282733206-32107" daemon prio=5 tid=32107 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "qtp1387697241-30537" daemon prio=5 tid=30537 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "pool-3446-thread-1" prio=5 tid=28881 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "IPC Server handler 5 on 50708" daemon prio=5 tid=28912 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data28)" daemon prio=5 tid=34496 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@6ffc0448" daemon prio=5 tid=28896 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:451) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data45)" daemon prio=5 tid=34514 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1)" daemon prio=5 tid=34468 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data55)" daemon prio=5 tid=34524 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data62)" daemon prio=5 tid=34531 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 1 on 50708" daemon prio=5 tid=28908 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data60)" daemon prio=5 tid=34529 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data42)" daemon prio=5 tid=34511 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "Attach Listener" daemon prio=9 tid=27524 runnable java.lang. Thread .State: RUNNABLE "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data87)" daemon prio=5 tid=34557 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data10)" daemon prio=5 tid=34478 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data67)" daemon prio=5 tid=34536 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data61)" daemon prio=5 tid=34530 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 6 on 50708" daemon prio=5 tid=28913 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data63)" daemon prio=5 tid=34532 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@7ef2b559" daemon prio=5 tid=28922 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:3881) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data44)" daemon prio=5 tid=34513 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "pool-3457-thread-1" prio=5 tid=32100 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "org.eclipse.jetty.server.session.HashSessionManager@6a64387cTimer" daemon prio=5 tid=32109 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "qtp490528317-28888" daemon prio=5 tid=28888 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "qtp490528317-28887" daemon prio=5 tid=28887 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang. Thread .run( Thread .java:745) "IPC Server handler 3 on 50723" daemon prio=5 tid=32125 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "CacheReplicationMonitor(1664877657)" prio=5 tid=28923 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data4)" daemon prio=5 tid=34472 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.eclipse.jetty.server.session.HashSessionManager@5bb89342Timer" daemon prio=5 tid=28892 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data36)" daemon prio=5 tid=34505 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 0 on 50708" daemon prio=5 tid=28907 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data12)" daemon prio=5 tid=34480 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data15)" daemon prio=5 tid=34483 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3)" daemon prio=5 tid=34471 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data64)" daemon prio=5 tid=34533 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data18)" daemon prio=5 tid=34486 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "nioEventLoopGroup-22-1" prio=10 tid=30541 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:621) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:309) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:703) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang. Thread .run( Thread .java:745) "Socket Reader #1 for port 50718" prio=5 tid=30544 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1088) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1067) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data35)" daemon prio=5 tid=34504 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=5 tid=64 in Object .wait() java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3698) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data47)" daemon prio=5 tid=34516 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.eclipse.jetty.server.session.HashSessionManager@1c38d141Timer" daemon prio=5 tid=32111 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "DecommissionMonitor-0" daemon prio=5 tid=28905 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data9)" daemon prio=5 tid=34477 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 9 on 50723" daemon prio=5 tid=32131 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "org.eclipse.jetty.server.session.HashSessionManager@2eaae4c6Timer" daemon prio=5 tid=28890 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data49)" daemon prio=5 tid=34518 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 2 on 50723" daemon prio=5 tid=32124 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "IPC Server handler 6 on 50723" daemon prio=5 tid=32128 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "IPC Server listener on 50708" daemon prio=5 tid=28898 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener.run(Server.java:1150) " Thread -26471" prio=5 tid=34580 runnable java.lang. Thread .State: RUNNABLE at org.eclipse.jetty.util.thread.QueuedThreadPool.doStop(QueuedThreadPool.java:181) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) at org.eclipse.jetty.server.Server.doStop(Server.java:482) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1071) at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.stop(NameNodeHttpServer.java:243) at org.apache.hadoop.hdfs.server.namenode.NameNode.stopHttpServer(NameNode.java:884) at org.apache.hadoop.hdfs.server.namenode.NameNode.stopCommonServices(NameNode.java:841) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1009) at org.apache.hadoop.hdfs.server.namenode.NameNode.stopAtException(NameNode.java:960) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:949) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:919) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1636) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1263) at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1032) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:907) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:839) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:491) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:450) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.startDFSCluster(TestDataNodeHotSwapVolumes.java:137) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.startDFSCluster(TestDataNodeHotSwapVolumes.java:108) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:421) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data39)" daemon prio=5 tid=34508 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@6b2cbc8f" daemon prio=5 tid=28921 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:3794) at java.lang. Thread .run( Thread .java:745) "qtp1387697241-30530" daemon prio=5 tid=30530 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745) "IPC Server handler 4 on 50723" daemon prio=5 tid=32126 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data54)" daemon prio=5 tid=34523 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data80)" daemon prio=5 tid=34550 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 3 on 50708" daemon prio=5 tid=28910 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "org.apache.hadoop.util.JvmPauseMonitor$Monitor@7e0a2aa" daemon prio=5 tid=28880 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192) at java.lang. Thread .run( Thread .java:745) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data56)" daemon prio=5 tid=34525 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "IPC Server handler 5 on 50723" daemon prio=5 tid=32127 timed_waiting java.lang. Thread .State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:218) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2639) "VolumeScannerThread(/Users/lei/work/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data82)" daemon prio=5 tid=34552 timed_waiting java.lang. Thread .State: TIMED_WAITING at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:620) "qtp282733206-32103-acceptor-0@59ab2598-ServerConnector@2635a54a{HTTP/1.1,[http/1.1]}{localhost:50721}" daemon prio=3 tid=32103 runnable java.lang. Thread .State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang. Thread .run( Thread .java:745)
        Hide
        manojg Manoj Govindassamy added a comment -

        Thanks for the review Yiqun Lin. Updated the patch with above changes.

        Show
        manojg Manoj Govindassamy added a comment - Thanks for the review Yiqun Lin . Updated the patch with above changes.
        Hide
        linyiqun Yiqun Lin added a comment - - edited

        Thanks Manoj Govindassamy for updating the patch. The latest patch looks pretty good now. Two minor comments:

        • Can we define a var named DEFAULT_STORAGES_PER_DATANODE to replace 2? That will be easily understood.
           private void startDFSCluster(int numNameNodes, int numDataNodes)
               throws IOException {
        +    startDFSCluster(numNameNodes, numDataNodes, 2);
        +  }
        
        • The delay time of addVolume is a little short. I tested your patch in my local many times, the most of the results were still passed with the ArrayList.
        if (r.nextInt(10) > 4) {
          int s = r.nextInt(10) + 1;
          Thread.sleep(s);
         }
        

        I increased the delay here, change Thread.sleep(s) to Thread.sleep(s * 100), then the tests runs as we expected,

        +1 once these are addressed. Thanks.

        Show
        linyiqun Yiqun Lin added a comment - - edited Thanks Manoj Govindassamy for updating the patch. The latest patch looks pretty good now. Two minor comments: Can we define a var named DEFAULT_STORAGES_PER_DATANODE to replace 2 ? That will be easily understood. private void startDFSCluster( int numNameNodes, int numDataNodes) throws IOException { + startDFSCluster(numNameNodes, numDataNodes, 2); + } The delay time of addVolume is a little short. I tested your patch in my local many times, the most of the results were still passed with the ArrayList . if (r.nextInt(10) > 4) { int s = r.nextInt(10) + 1; Thread .sleep(s); } I increased the delay here, change Thread.sleep(s) to Thread.sleep(s * 100) , then the tests runs as we expected, +1 once these are addressed. Thanks.
        Hide
        manojg Manoj Govindassamy added a comment -

        Attached v02 patch to address the following:
        1. Added a new test TestDataNodeHotSwapVolumes#testAddVolumesConcurrently to recreate the problem of ConcurrentModificationException during addVolume and fail when run without the proposed fix.
        2. HDFS-11267 has been filed to track the checkstyle issue.

        PS: testAddVolumesConcurrently currently has 40 new volumes to be added concurrently. Without fix, this test fails most of the times. You might want to increase the volume count to a higher number to make it fail all the times.

        Lei (Eddy) Xu, Yiqun Lin, can you please take a look at the patch ?

        Show
        manojg Manoj Govindassamy added a comment - Attached v02 patch to address the following: 1. Added a new test TestDataNodeHotSwapVolumes#testAddVolumesConcurrently to recreate the problem of ConcurrentModificationException during addVolume and fail when run without the proposed fix. 2. HDFS-11267 has been filed to track the checkstyle issue. PS: testAddVolumesConcurrently currently has 40 new volumes to be added concurrently. Without fix, this test fails most of the times. You might want to increase the volume count to a higher number to make it fail all the times. Lei (Eddy) Xu , Yiqun Lin , can you please take a look at the patch ?
        Hide
        manojg Manoj Govindassamy added a comment -

        sure, will give a try on this and upload a new patch soon. thanks.

        Show
        manojg Manoj Govindassamy added a comment - sure, will give a try on this and upload a new patch soon. thanks.
        Hide
        manojg Manoj Govindassamy added a comment -

        Sure, this is doable. I was inducing a delay in refreshVolumes to simulate race. I can try something with Mockito to simulate my testing.

        Show
        manojg Manoj Govindassamy added a comment - Sure, this is doable. I was inducing a delay in refreshVolumes to simulate race. I can try something with Mockito to simulate my testing.
        Hide
        linyiqun Yiqun Lin added a comment -

        I looked into this again, we can create some threads to execute addVolumes concurrently here. This will be easy to test this case.

        Show
        linyiqun Yiqun Lin added a comment - I looked into this again, we can create some threads to execute addVolumes concurrently here. This will be easy to test this case.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks for providing the patch, Manoj Govindassamy. The patch almost looks good to me, only some comments for your test. I ran the test in my local, it can run well without using CopyOnWriteArrayList. Can you add the test that can trigger the CME here as you mentioned in your comments? I think this will be good to test for this. Thanks.

        Show
        linyiqun Yiqun Lin added a comment - Thanks for providing the patch, Manoj Govindassamy . The patch almost looks good to me, only some comments for your test. I ran the test in my local, it can run well without using CopyOnWriteArrayList . Can you add the test that can trigger the CME here as you mentioned in your comments? I think this will be good to test for this. Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 14m 47s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 43s trunk passed
        +1 javadoc 0m 37s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 150 unchanged - 2 fixed = 151 total (was 152)
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 36s the patch passed
        +1 unit 70m 23s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 25s The patch does not generate ASF License warnings.
        96m 51s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11251
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844289/HDFS-11251.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3e521a785221 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8b042bc
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17925/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17925/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17925/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 47s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 37s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 150 unchanged - 2 fixed = 151 total (was 152) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 36s the patch passed +1 unit 70m 23s hadoop-hdfs in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 96m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11251 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844289/HDFS-11251.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3e521a785221 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b042bc Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17925/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17925/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17925/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        manojg Manoj Govindassamy added a comment -

        Attaching v01 patch to address the following.
        1. Made Storage#storageDirs a CopyOnWriteArrayList instead of a normal ArrayList so that concurrent iterators and writers don't end up in ConcurrentModificationException. In this case, substitution with CopyOnWriteArrayList is ok and should not cause any peformance degradation as Volume add and removal are not frequent operations.
        2. Tests updated to add more volumes during write and to expose the race condition. On a private branched I recreated the original CME problem with this test along with some delays introduced in Storage#containsStorageDir

        This patch will throw a CheckStyle issue on Storage#storageDirs not being private. But, this checkstyle issue existed even before the patch and touching the same line again will cause the checkstyle to show up again. Will file a new jira to cleanup the specifier for Storage#storageDirs and its access patterns in the derived classes.

        Lei (Eddy) Xu, Xiao Chen, Yiqun Lin, Can you please review the patch ?

        Show
        manojg Manoj Govindassamy added a comment - Attaching v01 patch to address the following. 1. Made Storage#storageDirs a CopyOnWriteArrayList instead of a normal ArrayList so that concurrent iterators and writers don't end up in ConcurrentModificationException . In this case, substitution with CopyOnWriteArrayList is ok and should not cause any peformance degradation as Volume add and removal are not frequent operations. 2. Tests updated to add more volumes during write and to expose the race condition. On a private branched I recreated the original CME problem with this test along with some delays introduced in Storage#containsStorageDir This patch will throw a CheckStyle issue on Storage#storageDirs not being private. But, this checkstyle issue existed even before the patch and touching the same line again will cause the checkstyle to show up again. Will file a new jira to cleanup the specifier for Storage#storageDirs and its access patterns in the derived classes. Lei (Eddy) Xu , Xiao Chen , Yiqun Lin , Can you please review the patch ?
        Hide
        manojg Manoj Govindassamy added a comment -

        Thanks Yiqun Lin.

        Yes, I do see the addVolume volume operation from the stack trace. But, that volume adding thread shown in the stack trace is just in the DataStorage#prepareVolume phase and it is only traversing the storageDirs in Storage#containsStorageDir. That is, there is yet another thread which is mutating the same ArrayList around the same time when the first Volume Add was happening.

        Looked at the test code again and there are 2 Volume Add happening as part of the test. As you said, these Volume add operations are run in Executors via FutureTask and hence these are submitted in quick succession and are running parallely and are mutating the same storageDirs list.

        I am able to recreate the ConcurrentModificationException by following 2 ways:

        • Running a new Thread which continuously runs a read operation on the StorageDirs (like listStorageDirectories) and then having Volume add in parallel, OR
        • Adding 10 volumes with a small delay between each other, so that each of volume add's list traversing will trip over previous volume add list modification.

        My fix proposal is to create Storage#storageDirs as new CopyOnWriteArrayList<StorageDirectory>(); Will test out and submit a patch.

        Show
        manojg Manoj Govindassamy added a comment - Thanks Yiqun Lin . Yes, I do see the addVolume volume operation from the stack trace. But, that volume adding thread shown in the stack trace is just in the DataStorage#prepareVolume phase and it is only traversing the storageDirs in Storage#containsStorageDir . That is, there is yet another thread which is mutating the same ArrayList around the same time when the first Volume Add was happening. Looked at the test code again and there are 2 Volume Add happening as part of the test. As you said, these Volume add operations are run in Executors via FutureTask and hence these are submitted in quick succession and are running parallely and are mutating the same storageDirs list. I am able to recreate the ConcurrentModificationException by following 2 ways: Running a new Thread which continuously runs a read operation on the StorageDirs (like listStorageDirectories) and then having Volume add in parallel, OR Adding 10 volumes with a small delay between each other, so that each of volume add's list traversing will trip over previous volume add list modification. My fix proposal is to create Storage#storageDirs as new CopyOnWriteArrayList<StorageDirectory>() ; Will test out and submit a patch.
        Hide
        linyiqun Yiqun Lin added a comment - - edited

        Thanks Manoj Govindassamy for the analysis. I think that's the reason of the failure case. Here the add volume or remove volume is a asynchronized operation so there is a chance to lead the CME.

        Want to look at logs to find the parallel operations on the storageDir

        Here it's the addVolume operation caused this as you can see the stack info that Jason Lowe provided above. Hope this can help you.

        org.apache.hadoop.conf.ReconfigurationException: Could not change property dfs.datanode.data.dir from '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4' to '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4'
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.refreshVolumes(DataNode.java:777)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:532)
        	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.addVolumes(TestDataNodeHotSwapVolumes.java:310)
        	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:404)
        
        Show
        linyiqun Yiqun Lin added a comment - - edited Thanks Manoj Govindassamy for the analysis. I think that's the reason of the failure case. Here the add volume or remove volume is a asynchronized operation so there is a chance to lead the CME. Want to look at logs to find the parallel operations on the storageDir Here it's the addVolume operation caused this as you can see the stack info that Jason Lowe provided above. Hope this can help you. org.apache.hadoop.conf.ReconfigurationException: Could not change property dfs.datanode.data.dir from '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4' to '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4' at org.apache.hadoop.hdfs.server.datanode.DataNode.refreshVolumes(DataNode.java:777) at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:532) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.addVolumes(TestDataNodeHotSwapVolumes.java:310) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:404)
        Hide
        manojg Manoj Govindassamy added a comment -

        Hi Jason Lowe,

        Amny chance you have the full log file for this failure case ? Would like to take a look.

        Storage#storageDirs is not a concurrent list. So, parallel addition or removal of volumes with list iteration can throw ConcurrentModificationException. Want to look at logs to find the parallel operations on the storageDir. One of the fixes could be building Storage#storageDirs as a Collections.synchronizedList(..). Other could be locking down the modification operations and iteration operations.

        Show
        manojg Manoj Govindassamy added a comment - Hi Jason Lowe , Amny chance you have the full log file for this failure case ? Would like to take a look. Storage#storageDirs is not a concurrent list. So, parallel addition or removal of volumes with list iteration can throw ConcurrentModificationException. Want to look at logs to find the parallel operations on the storageDir. One of the fixes could be building Storage#storageDirs as a Collections.synchronizedList(..). Other could be locking down the modification operations and iteration operations.
        Hide
        jlowe Jason Lowe added a comment -

        The test failed with this stacktrace:

        org.apache.hadoop.conf.ReconfigurationException: Could not change property dfs.datanode.data.dir from '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4' to '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4'
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.refreshVolumes(DataNode.java:777)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:532)
        	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.addVolumes(TestDataNodeHotSwapVolumes.java:310)
        	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:404)
        
        

        In the test output I found a CME which appears to be the cause. If so, it'd be nice if ReconfigurationException relayed the exception that caused the failure.

        2016-12-15 00:33:21,848 [pool-239-thread-2] INFO  impl.FsDatasetImpl (FsVolumeList.java:addVolume(320)) - Added new volume: DS-6c2d1743-ee6f-4011-8042-b47d45d5279b
        2016-12-15 00:33:21,848 [pool-239-thread-2] INFO  impl.FsDatasetImpl (FsDatasetImpl.java:addVolume(494)) - Added volume - [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4, StorageType: DISK
        2016-12-15 00:33:21,851 [Thread-1888] ERROR datanode.DataNode (DataNode.java:refreshVolumes(764)) - Failed to add volume: [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3
        java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
        	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.refreshVolumes(DataNode.java:750)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:532)
        	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.addVolumes(TestDataNodeHotSwapVolumes.java:310)
        	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:404)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
        Caused by: java.util.ConcurrentModificationException
        	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
        	at java.util.ArrayList$Itr.next(ArrayList.java:851)
        	at org.apache.hadoop.hdfs.server.common.Storage.containsStorageDir(Storage.java:999)
        	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:220)
        	at org.apache.hadoop.hdfs.server.datanode.DataStorage.prepareVolume(DataStorage.java:332)
        	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.addVolume(FsDatasetImpl.java:455)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode$2.call(DataNode.java:737)
        	at org.apache.hadoop.hdfs.server.datanode.DataNode$2.call(DataNode.java:733)
        	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        	at java.lang.Thread.run(Thread.java:745)
        
        Show
        jlowe Jason Lowe added a comment - The test failed with this stacktrace: org.apache.hadoop.conf.ReconfigurationException: Could not change property dfs.datanode.data.dir from '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4' to '[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data1,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data2,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3,[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4' at org.apache.hadoop.hdfs.server.datanode.DataNode.refreshVolumes(DataNode.java:777) at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:532) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.addVolumes(TestDataNodeHotSwapVolumes.java:310) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:404) In the test output I found a CME which appears to be the cause. If so, it'd be nice if ReconfigurationException relayed the exception that caused the failure. 2016-12-15 00:33:21,848 [pool-239-thread-2] INFO impl.FsDatasetImpl (FsVolumeList.java:addVolume(320)) - Added new volume: DS-6c2d1743-ee6f-4011-8042-b47d45d5279b 2016-12-15 00:33:21,848 [pool-239-thread-2] INFO impl.FsDatasetImpl (FsDatasetImpl.java:addVolume(494)) - Added volume - [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data4, StorageType: DISK 2016-12-15 00:33:21,851 [Thread-1888] ERROR datanode.DataNode (DataNode.java:refreshVolumes(764)) - Failed to add volume: [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/4/dfs/data/data3 java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hdfs.server.datanode.DataNode.refreshVolumes(DataNode.java:750) at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:532) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.addVolumes(TestDataNodeHotSwapVolumes.java:310) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testAddVolumesDuringWrite(TestDataNodeHotSwapVolumes.java:404) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) at java.util.ArrayList$Itr.next(ArrayList.java:851) at org.apache.hadoop.hdfs.server.common.Storage.containsStorageDir(Storage.java:999) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:220) at org.apache.hadoop.hdfs.server.datanode.DataStorage.prepareVolume(DataStorage.java:332) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.addVolume(FsDatasetImpl.java:455) at org.apache.hadoop.hdfs.server.datanode.DataNode$2.call(DataNode.java:737) at org.apache.hadoop.hdfs.server.datanode.DataNode$2.call(DataNode.java:733) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

          People

          • Assignee:
            manojg Manoj Govindassamy
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development