Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14306

TestLocalFileSystem tests have very low timeouts

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Most tests have a timeout of 1 second, which is much too low, especially if there is a spinning disk involved.

      1. HADOOP-14306.001.patch
        4 kB
        Eric Badger
      2. HADOOP-14306.002.patch
        5 kB
        Eric Badger
      3. HADOOP-14306.003.patch
        5 kB
        Eric Badger

        Activity

        Hide
        ebadger Eric Badger added a comment -

        Getting rid of timeouts for tests. They aren't really necessary since the code isn't doing anything but straight-line code.

        Show
        ebadger Eric Badger added a comment - Getting rid of timeouts for tests. They aren't really necessary since the code isn't doing anything but straight-line code.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 14m 5s trunk passed
        +1 compile 17m 25s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 1m 5s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        +1 findbugs 1m 32s trunk passed
        +1 javadoc 0m 51s trunk passed
        +1 mvninstall 0m 41s the patch passed
        +1 compile 13m 15s the patch passed
        +1 javac 13m 15s the patch passed
        +1 checkstyle 0m 35s the patch passed
        +1 mvnsite 1m 5s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 49s the patch passed
        -1 unit 8m 45s hadoop-common in the patch failed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        65m 56s



        Reason Tests
        Failed junit tests hadoop.security.TestKDiag



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:612578f
        JIRA Issue HADOOP-14306
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863373/HADOOP-14306.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 831115742aa2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0cab572
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12100/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12100/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12100/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 5s trunk passed +1 compile 17m 25s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 51s trunk passed +1 mvninstall 0m 41s the patch passed +1 compile 13m 15s the patch passed +1 javac 13m 15s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 8m 45s hadoop-common in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 65m 56s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14306 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863373/HADOOP-14306.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 831115742aa2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12100/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12100/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12100/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        aw Allen Wittenauer added a comment -

        The timeouts are there in case the test hangs (e.g., disk failure). What will prevent maven getting stuck and eating a slot on the build server if there is no timeout?

        Show
        aw Allen Wittenauer added a comment - The timeouts are there in case the test hangs (e.g., disk failure). What will prevent maven getting stuck and eating a slot on the build server if there is no timeout?
        Hide
        stevel@apache.org Steve Loughran added a comment -

        -1

        you've actually turned off all the tests

        Also, like AW notes, hadoop tests mandate a timeout, though we nowadays use per-suite timeout for easier maintenance. Look at AbstractFSContractTestBase to see how to use a test rule for that.

        Show
        stevel@apache.org Steve Loughran added a comment - -1 you've actually turned off all the tests Also, like AW notes, hadoop tests mandate a timeout, though we nowadays use per-suite timeout for easier maintenance. Look at AbstractFSContractTestBase to see how to use a test rule for that.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        +Allen: maven itself has test run timeouts.

        Show
        stevel@apache.org Steve Loughran added a comment - +Allen: maven itself has test run timeouts.
        Hide
        ebadger Eric Badger added a comment -

        Allen Wittenauer, Steve Loughran, thanks for the review/advice. My initial thinking was to get rid of the specific timeouts on the tests so that they would just use the maven-wide timeout. But I agree that a test suite timeout is a better solution. Uploading a new patch that gets rid of the specific timeouts on each test, but adds a test suite timeout of 60 seconds. This is a very conservative timeout as the largest timeout of the tests beforehand was 10 seconds. So let me know if you think that it should be more aggressive.

        Show
        ebadger Eric Badger added a comment - Allen Wittenauer , Steve Loughran , thanks for the review/advice. My initial thinking was to get rid of the specific timeouts on the tests so that they would just use the maven-wide timeout. But I agree that a test suite timeout is a better solution. Uploading a new patch that gets rid of the specific timeouts on each test, but adds a test suite timeout of 60 seconds. This is a very conservative timeout as the largest timeout of the tests beforehand was 10 seconds. So let me know if you think that it should be more aggressive.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 16s trunk passed
        +1 compile 15m 34s trunk passed
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        +1 findbugs 1m 26s trunk passed
        +1 javadoc 0m 49s trunk passed
        +1 mvninstall 0m 38s the patch passed
        +1 compile 13m 56s the patch passed
        +1 javac 13m 56s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 1m 2s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 34s the patch passed
        +1 javadoc 0m 50s the patch passed
        -1 unit 7m 56s hadoop-common in the patch failed.
        +1 asflicense 0m 37s The patch does not generate ASF License warnings.
        62m 44s



        Reason Tests
        Failed junit tests hadoop.security.TestRaceWhenRelogin
          hadoop.security.TestKDiag



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:612578f
        JIRA Issue HADOOP-14306
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863468/HADOOP-14306.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f6c2520f7bff 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0cab572
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 16s trunk passed +1 compile 15m 34s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 49s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 13m 56s the patch passed +1 javac 13m 56s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 50s the patch passed -1 unit 7m 56s hadoop-common in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 62m 44s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin   hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14306 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863468/HADOOP-14306.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f6c2520f7bff 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        the trouble with the maven timeout is that it just kills the whole JVM. As the junit test runner builds up the XML test report in memory, killing the JVM loses the entire log of the test run which failed out. This is generally considered of limited value when trying to work out why a test failed

        Show
        stevel@apache.org Steve Loughran added a comment - the trouble with the maven timeout is that it just kills the whole JVM. As the junit test runner builds up the XML test report in memory, killing the JVM loses the entire log of the test run which failed out. This is generally considered of limited value when trying to work out why a test failed
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch, Eric!

        Nit: I don't see the point of having a protected method for getting the test timeout value. For AbstractFSContractTestBase it makes sense since the whole point of that class is to be subclassed by the "real" tests. In this case nothing is going to subclass it, so it's just extra boilerplate with no purpose.

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch, Eric! Nit: I don't see the point of having a protected method for getting the test timeout value. For AbstractFSContractTestBase it makes sense since the whole point of that class is to be subclassed by the "real" tests. In this case nothing is going to subclass it, so it's just extra boilerplate with no purpose.
        Hide
        ebadger Eric Badger added a comment -

        Thanks, Jason Lowe for the review! That makes sense. Uploading a new patch where I got rid of the protected method and just created the timeout straight up

        Show
        ebadger Eric Badger added a comment - Thanks, Jason Lowe for the review! That makes sense. Uploading a new patch where I got rid of the protected method and just created the timeout straight up
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 15m 0s trunk passed
        +1 compile 19m 35s trunk passed
        +1 checkstyle 0m 43s trunk passed
        +1 mvnsite 1m 19s trunk passed
        +1 mvneclipse 0m 23s trunk passed
        -1 findbugs 2m 6s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings.
        +1 javadoc 1m 3s trunk passed
        +1 mvninstall 1m 4s the patch passed
        +1 compile 17m 48s the patch passed
        +1 javac 17m 48s the patch passed
        +1 checkstyle 0m 39s the patch passed
        +1 mvnsite 1m 7s the patch passed
        +1 mvneclipse 0m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 38s the patch passed
        +1 javadoc 0m 47s the patch passed
        +1 unit 7m 53s hadoop-common in the patch passed.
        +1 asflicense 0m 42s The patch does not generate ASF License warnings.
        74m 34s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ac17dc
        JIRA Issue HADOOP-14306
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865428/HADOOP-14306.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d3d9f0bdd347 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9460721
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12203/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12203/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12203/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 0s trunk passed +1 compile 19m 35s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 23s trunk passed -1 findbugs 2m 6s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. +1 javadoc 1m 3s trunk passed +1 mvninstall 1m 4s the patch passed +1 compile 17m 48s the patch passed +1 javac 17m 48s the patch passed +1 checkstyle 0m 39s the patch passed +1 mvnsite 1m 7s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 38s the patch passed +1 javadoc 0m 47s the patch passed +1 unit 7m 53s hadoop-common in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 74m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HADOOP-14306 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865428/HADOOP-14306.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d3d9f0bdd347 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9460721 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12203/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12203/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12203/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        +1 lgtm. I'll commit this later today if there are no objections.

        Show
        jlowe Jason Lowe added a comment - +1 lgtm. I'll commit this later today if there are no objections.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks to Eric for the contribution and to Steve and Allen for additional review! I committed this to trunk, branch-2, and branch-2.8.

        Show
        jlowe Jason Lowe added a comment - Thanks to Eric for the contribution and to Steve and Allen for additional review! I committed this to trunk, branch-2, and branch-2.8.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11675 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11675/)
        HADOOP-14306. TestLocalFileSystem tests have very low timeouts. (jlowe: rev cedaf4cab9d5013acd559dcb92d4976823ccdf85)

        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11675 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11675/ ) HADOOP-14306 . TestLocalFileSystem tests have very low timeouts. (jlowe: rev cedaf4cab9d5013acd559dcb92d4976823ccdf85) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java

          People

          • Assignee:
            ebadger Eric Badger
            Reporter:
            ebadger Eric Badger
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development