Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13459

hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Within hadoop-azure, we have the NativeAzureFileSystemBaseTest abstract class, which defines setup and teardown to handle the Azure storage account and also defines multiple test cases. This class originally was contributed to provide a layer of indirection for running the same test cases in live mode or mock mode: TestNativeAzureFileSystemLive and TestNativeAzureFileSystemMocked. It appears that since then, we created multiple new test suites that subclassed NativeAzureFileSystemBaseTest for the benefit of getting the common setup and teardown code, but also with the side effect of running the inherited test cases repeatedly. This is a significant factor in the overall execution time of the hadoop-azure tests.

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          This patch refactors the tests to introduce a new abstract class: AbstractWasbTestBase. The vast majority of test suites can subclass this to get just the setup and teardown logic without accidentally re-running test cases.

          Here is some information on running time for the impacted test suites before and after the patch.

          Before:
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Tests run: 62, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 325.178 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 316.563 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 392.583 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 284.985 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 300.001 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 296.564 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.994 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked

          After:
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.088 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.198 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.855 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.6 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.944 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 307.217 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.915 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked

          Show
          cnauroth Chris Nauroth added a comment - This patch refactors the tests to introduce a new abstract class: AbstractWasbTestBase . The vast majority of test suites can subclass this to get just the setup and teardown logic without accidentally re-running test cases. Here is some information on running time for the impacted test suites before and after the patch. Before: Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Tests run: 62, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 325.178 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 316.563 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 392.583 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 284.985 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 300.001 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 296.564 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.994 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked After: Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.088 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.198 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.855 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.6 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.944 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 307.217 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.915 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          It is great to reduce the test run time.

          +1 patch looks good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - It is great to reduce the test run time. +1 patch looks good.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
          +1 mvninstall 7m 20s trunk passed
          +1 compile 0m 18s trunk passed
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 0m 24s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 15s the patch passed
          +1 javac 0m 15s the patch passed
          -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 6 new + 91 unchanged - 6 fixed = 97 total (was 97)
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 30s the patch passed
          +1 javadoc 0m 9s the patch passed
          +1 unit 1m 20s hadoop-azure in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          14m 10s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821725/HADOOP-13459.001.patch
          JIRA Issue HADOOP-13459
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 946c710e6fac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d28c2d9
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10159/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10159/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10159/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. +1 mvninstall 7m 20s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 6 new + 91 unchanged - 6 fixed = 97 total (was 97) +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 9s the patch passed +1 unit 1m 20s hadoop-azure in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 14m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821725/HADOOP-13459.001.patch JIRA Issue HADOOP-13459 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 946c710e6fac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d28c2d9 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10159/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10159/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10159/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk, branch-2 and branch-2.8. Nicholas, thank you for the code review.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-2 and branch-2.8. Nicholas, thank you for the code review.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10199 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10199/)
          HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing (cnauroth: rev 4ef1bc0bdb6b2c907431492e4c8395035fa96c84)

          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsExceptionHandlingMultiThreaded.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAtomicRenameDirList.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAppend.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestBase.java
          • hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationExceptionHandling.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10199 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10199/ ) HADOOP-13459 . hadoop-azure runs several test cases repeatedly, causing (cnauroth: rev 4ef1bc0bdb6b2c907431492e4c8395035fa96c84) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationsExceptionHandlingMultiThreaded.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAtomicRenameDirList.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAppend.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestBase.java hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationExceptionHandling.java

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development