Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9744

TestDirectoryScanner#testThrottling occasionally time out after 300 seconds

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
    • Environment:

      Jenkins

    • Hadoop Flags:
      Reviewed

      Description

      I have seen quite a few test failures in TestDirectoryScanner#testThrottling.
      https://builds.apache.org/job/Hadoop-Hdfs-trunk/2793/testReport/org.apache.hadoop.hdfs.server.datanode/TestDirectoryScanner/testThrottling/

      Looking at the log, it does not look like the test got stucked. On my local machine, this test took 219 seconds. It is likely that this test takes more than 300 seconds to complete on a busy jenkins slave. I think it is reasonable to set a longer time out value, or reduce the number of blocks to reduce the duration of the test.

      Error Message

      test timed out after 300000 milliseconds
      

      Stacktrace

      java.lang.Exception: test timed out after 300000 milliseconds
      	at java.lang.Object.wait(Native Method)
      	at java.lang.Object.wait(Object.java:503)
      	at org.apache.hadoop.hdfs.DataStreamer.waitAndQueuePacket(DataStreamer.java:804)
      	at org.apache.hadoop.hdfs.DFSOutputStream.enqueueCurrentPacket(DFSOutputStream.java:423)
      	at org.apache.hadoop.hdfs.DFSOutputStream.enqueueCurrentPacketFull(DFSOutputStream.java:432)
      	at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:418)
      	at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:217)
      	at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:125)
      	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:111)
      	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
      	at java.io.DataOutputStream.write(DataOutputStream.java:107)
      	at org.apache.hadoop.hdfs.DFSTestUtil.createFile(DFSTestUtil.java:418)
      	at org.apache.hadoop.hdfs.DFSTestUtil.createFile(DFSTestUtil.java:376)
      	at org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner.createFile(TestDirectoryScanner.java:108)
      	at org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner.testThrottling(TestDirectoryScanner.java:584)
      

        Activity

        Hide
        templedf Daniel Templeton added a comment -

        Bump the timeout (or just remove it). If you drop the block count, you'll compromise the test.

        Show
        templedf Daniel Templeton added a comment - Bump the timeout (or just remove it). If you drop the block count, you'll compromise the test.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Ok. I just feel a bit unfortunate that a single test can take more than 5 minutes to complete. Is there anyway to reduce the test time without compromising the test?

        Show
        jojochuang Wei-Chiu Chuang added a comment - Ok. I just feel a bit unfortunate that a single test can take more than 5 minutes to complete. Is there anyway to reduce the test time without compromising the test?
        Hide
        templedf Daniel Templeton added a comment -

        The problem is that there have to be enough blocks to make the scanning take long enough to be able to measure the throttling. Since we can't predict where the test will run, it's hard to pin that number. Too high makes the test take forever. Too low and the test breaks. Feel free to try to tune the block count, but understand that you're kinda stuck between a rock and a hard place.

        Show
        templedf Daniel Templeton added a comment - The problem is that there have to be enough blocks to make the scanning take long enough to be able to measure the throttling. Since we can't predict where the test will run, it's hard to pin that number. Too high makes the test take forever. Too low and the test breaks. Feel free to try to tune the block count, but understand that you're kinda stuck between a rock and a hard place.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Wei-Chiu Chuang for finding this. I agree comments from Daniel Templeton. We can just bump the timeout for TestDirectory#testThrottling. Attach a simple patch from me.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Wei-Chiu Chuang for finding this. I agree comments from Daniel Templeton . We can just bump the timeout for TestDirectory#testThrottling . Attach a simple patch from me.
        Hide
        templedf Daniel Templeton added a comment -

        LGTM

        Show
        templedf Daniel Templeton added a comment - LGTM
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 19s trunk passed
        +1 compile 0m 44s trunk passed with JDK v1.8.0_77
        +1 compile 0m 46s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 59s trunk passed
        +1 javadoc 1m 11s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 48s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.8.0_77
        +1 javac 0m 37s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_95
        +1 javac 0m 41s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 21s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95
        -1 unit 56m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
        -1 unit 53m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        136m 30s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          hadoop.hdfs.TestMissingBlocksAlert
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          hadoop.hdfs.TestHFlush



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797231/HDFS-9744.001.patch
        JIRA Issue HDFS-9744
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e48365963697 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b5d4c7d
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15195/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15195/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 19s trunk passed +1 compile 0m 44s trunk passed with JDK v1.8.0_77 +1 compile 0m 46s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 48s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_77 +1 javac 0m 37s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_95 +1 javac 0m 41s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 21s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95 -1 unit 56m 38s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 53m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 136m 30s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.TestMissingBlocksAlert JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797231/HDFS-9744.001.patch JIRA Issue HDFS-9744 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e48365963697 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b5d4c7d Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15195/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15195/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15195/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        walter.k.su Walter Su added a comment -

        +1. will commit shortly.

        Show
        walter.k.su Walter Su added a comment - +1. will commit shortly.
        Hide
        walter.k.su Walter Su added a comment -

        Committed to trunk, branch-2, branch-2.8. Thanks Daniel Templeton for the review. Thanks Wei-Chiu Chuang for the report. And thanks Yiqun Lin for the contribution!

        Show
        walter.k.su Walter Su added a comment - Committed to trunk, branch-2, branch-2.8. Thanks Daniel Templeton for the review. Thanks Wei-Chiu Chuang for the report. And thanks Yiqun Lin for the contribution!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9633 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9633/)
        HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out (waltersu4549: rev 90d1af141666ba12f18d44464cb6169c445d0902)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9633 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9633/ ) HDFS-9744 . TestDirectoryScanner#testThrottling occasionally time out (waltersu4549: rev 90d1af141666ba12f18d44464cb6169c445d0902) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java

          People

          • Assignee:
            linyiqun Yiqun Lin
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development