Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:

      Description

      The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend most of their time blocked on network I/O with the S3 back-end, but they don't saturate the bandwidth of the NIC. We can improve overall execution time by enabling parallel test execution.

      1. HADOOP-13113.004.patch
        14 kB
        Chris Nauroth
      2. HADOOP-13113.003.patch
        14 kB
        Chris Nauroth
      3. HADOOP-13113.002.patch
        10 kB
        Chris Nauroth
      4. HADOOP-13113.001.patch
        9 kB
        Chris Nauroth

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          I was only testing TestS3A*, so it's not the cross FS problem...something else

          Show
          stevel@apache.org Steve Loughran added a comment - I was only testing TestS3A*, so it's not the cross FS problem...something else
          Hide
          cnauroth Chris Nauroth added a comment -

          I should warn that I'm seeing errors here...

          Steve Loughran, here are my observations based on additional experiments.

          1. If I configure a dedicated bucket per file system (S3, S3N and S3A), then I consistently get a full pass on parallel test runs. This was the setup I was using when I developed the patch.
          2. All of my buckets are in AWS region US-west-2 (Oregon), which is very close to my laptop physically.
          3. If I configure the same bucket to be used by the test suites for all 3 file systems, then sometimes I see some flaky behavior. I don't know if this is due to data problems, such as perhaps S3N tests leaving bits behind that confuse S3A later, or if the increased activity on a shared bucket simply surfaces eventual consistency problems with greater likelihood.
          4. One test run surfaced a bug, and I have submitted a patch for that on HADOOP-13158.
          5. TestS3NContractRootDir#testListEmptyRootDirectory probably needs the same retry loop that we put in place for the same test in TestS3AContractRootDir.

          I'll continue to look for ways to stabilize this.

          Show
          cnauroth Chris Nauroth added a comment - I should warn that I'm seeing errors here... Steve Loughran , here are my observations based on additional experiments. If I configure a dedicated bucket per file system (S3, S3N and S3A), then I consistently get a full pass on parallel test runs. This was the setup I was using when I developed the patch. All of my buckets are in AWS region US-west-2 (Oregon), which is very close to my laptop physically. If I configure the same bucket to be used by the test suites for all 3 file systems, then sometimes I see some flaky behavior. I don't know if this is due to data problems, such as perhaps S3N tests leaving bits behind that confuse S3A later, or if the increased activity on a shared bucket simply surfaces eventual consistency problems with greater likelihood. One test run surfaced a bug, and I have submitted a patch for that on HADOOP-13158 . TestS3NContractRootDir#testListEmptyRootDirectory probably needs the same retry loop that we put in place for the same test in TestS3AContractRootDir . I'll continue to look for ways to stabilize this.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I should warn that I'm seeing errors here; I think inconsistencies may be surfing —but I don't see why that should happen with parallel tests.

          Maybe if writes are taking time in the background after a close(), the longer that write takes to complete, the more chance the next operation may fail —and with multiple threads trying to use the same network connection, that's generating the load

          Show
          stevel@apache.org Steve Loughran added a comment - I should warn that I'm seeing errors here; I think inconsistencies may be surfing —but I don't see why that should happen with parallel tests. Maybe if writes are taking time in the background after a close(), the longer that write takes to complete, the more chance the next operation may fail —and with multiple threads trying to use the same network connection, that's generating the load
          Hide
          stevel@apache.org Steve Loughran added a comment -

          LGTM

          +1

          Show
          stevel@apache.org Steve Loughran added a comment - LGTM +1
          Hide
          cnauroth Chris Nauroth added a comment -

          Thank you, Steve. Here is patch v004 with the doc change and a minor rebase after the recent commits.

          Show
          cnauroth Chris Nauroth added a comment - Thank you, Steve. Here is patch v004 with the doc change and a minor rebase after the recent commits.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          oh, can you tack in a couple of lines in the aws index about how to enable this? thanks

          Show
          stevel@apache.org Steve Loughran added a comment - oh, can you tack in a couple of lines in the aws index about how to enable this? thanks
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1

          It's pretty painful to look at what you've had to do regarding test setup, but I don't see any other workaround. And JS is built into Java6+, so it'll work everywhere

          Show
          stevel@apache.org Steve Loughran added a comment - +1 It's pretty painful to look at what you've had to do regarding test setup, but I don't see any other workaround. And JS is built into Java6+, so it'll work everywhere
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          +1 mvninstall 6m 45s trunk passed
          +1 compile 0m 11s trunk passed with JDK v1.8.0_91
          +1 compile 0m 14s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 18s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 9s the patch passed with JDK v1.8.0_91
          +1 javac 0m 9s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.7.0_95
          +1 javac 0m 11s the patch passed
          +1 checkstyle 0m 11s hadoop-tools/hadoop-aws: The patch generated 0 new + 1 unchanged - 10 fixed = 1 total (was 11)
          +1 mvnsite 0m 16s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 0m 38s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          12m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803376/HADOOP-13113.003.patch
          JIRA Issue HADOOP-13113
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 4a30a38e7487 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d971bf2
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9366/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9366/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 11s trunk passed with JDK v1.8.0_91 +1 compile 0m 14s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 14s the patch passed +1 compile 0m 9s the patch passed with JDK v1.8.0_91 +1 javac 0m 9s the patch passed +1 compile 0m 11s the patch passed with JDK v1.7.0_95 +1 javac 0m 11s the patch passed +1 checkstyle 0m 11s hadoop-tools/hadoop-aws: The patch generated 0 new + 1 unchanged - 10 fixed = 1 total (was 11) +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 12m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803376/HADOOP-13113.003.patch JIRA Issue HADOOP-13113 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 4a30a38e7487 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d971bf2 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9366/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9366/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching patch v003. This adds the rety loop on TestS3AContractRootDir#testListEmptyRootDirectory. I also took the opportunity to clear some Checkstyle nits in files I'm touching.

          Do we need to rely on a new system property, or could we just rely on the test class name being unique? that is this.getClass().getName() could provide the path?

          Currently the patch relies on overriding AbstractBondedFSContract#getTestPath as a convenient place to parameterize the test path per unique fork. This layer of the code doesn't have direct access to the test suite class. We could change the contract classes so that the test suite class needs to be passed in during construction, but then every test suite would need to be changed. Overall, I prefer what the current patch is doing, but I could be convinced otherwise if you feel strongly about it.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching patch v003. This adds the rety loop on TestS3AContractRootDir#testListEmptyRootDirectory . I also took the opportunity to clear some Checkstyle nits in files I'm touching. Do we need to rely on a new system property, or could we just rely on the test class name being unique? that is this.getClass().getName() could provide the path? Currently the patch relies on overriding AbstractBondedFSContract#getTestPath as a convenient place to parameterize the test path per unique fork. This layer of the code doesn't have direct access to the test suite class. We could change the contract classes so that the test suite class needs to be passed in during construction, but then every test suite would need to be changed. Overall, I prefer what the current patch is doing, but I could be convinced otherwise if you feel strongly about it.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          other than that though, LGTM.

          Do we need to rely on a new system property, or could we just rely on the test class name being unique? that is this.getClass().getName() could provide the path?

          Show
          stevel@apache.org Steve Loughran added a comment - other than that though, LGTM. Do we need to rely on a new system property, or could we just rely on the test class name being unique? that is this.getClass().getName() could provide the path?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I think our tests generate race conditions more than most other bits of code do, as they are repeatedly working on the same bits of the objects store and trying to do lots of small operations.

          Looks like you are hitting the delayed-listing-consistency problem which S3 has —here on a delete, rather than an addition.

          why not have that test not fail immediately if the root dir isn't empty, but sleep and spin for a bit until the dir empties up.

          Show
          stevel@apache.org Steve Loughran added a comment - I think our tests generate race conditions more than most other bits of code do, as they are repeatedly working on the same bits of the objects store and trying to do lots of small operations. Looks like you are hitting the delayed-listing-consistency problem which S3 has —here on a delete, rather than an addition. why not have that test not fail immediately if the root dir isn't empty, but sleep and spin for a bit until the dir empties up.
          Hide
          cnauroth Chris Nauroth added a comment -

          One interesting finding from repeated parallel test runs is that I'm seeing a greater occurrence of failures in TestS3AContractRootDir#testListEmptyRootDirectory, despite the fact that this is one of the test suites that I force to run in serial.

          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.085 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir)  Time elapsed: 3.041 sec  <<< FAILURE!
          java.lang.AssertionError: Deleted file: unexpectedly found s3a://cnauroth-test-aws-s3a/fork-2 as  S3AFileStatus{path=s3a://cnauroth-test-aws-s3a/fork-2; isDirectory=true; modification_time=0; access_time=0; owner=; group=; permission=rwxrwxrwx; isSymlink=false}
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:706)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:566)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:544)
          	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:134)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          

          I do see an intermittent failure of this test even without the parallel test patch, so it's not something introduced by this patch. I think the test is subject to failure because of the eventual consistency behavior of deletes against S3. Reviewing debug logs, in the failure case, I get the "Deleting fake empty directory" message, even though it's definitely not empty at this point. If I put a retry loop around the test, sometimes it fails first and then succeeds on the retry.

          It might be interesting to explore if the order of operations in S3AFileSystem#listStatus could be changed to help with this. Maybe the common prefixes could be analyzed first before deciding the key represents a fake empty directory. I'm not sure if it's completely solvable though, because any kind of S3 listing modification is subject to eventual consistency effects. I'm not planning to change anything related to this within the scope of the current patch though.

          Show
          cnauroth Chris Nauroth added a comment - One interesting finding from repeated parallel test runs is that I'm seeing a greater occurrence of failures in TestS3AContractRootDir#testListEmptyRootDirectory , despite the fact that this is one of the test suites that I force to run in serial. Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.085 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir) Time elapsed: 3.041 sec <<< FAILURE! java.lang.AssertionError: Deleted file: unexpectedly found s3a: //cnauroth-test-aws-s3a/fork-2 as S3AFileStatus{path=s3a://cnauroth-test-aws-s3a/fork-2; isDirectory= true ; modification_time=0; access_time=0; owner=; group=; permission=rwxrwxrwx; isSymlink= false } at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:706) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:566) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:544) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:134) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) I do see an intermittent failure of this test even without the parallel test patch, so it's not something introduced by this patch. I think the test is subject to failure because of the eventual consistency behavior of deletes against S3. Reviewing debug logs, in the failure case, I get the "Deleting fake empty directory" message, even though it's definitely not empty at this point. If I put a retry loop around the test, sometimes it fails first and then succeeds on the retry. It might be interesting to explore if the order of operations in S3AFileSystem#listStatus could be changed to help with this. Maybe the common prefixes could be analyzed first before deciding the key represents a fake empty directory. I'm not sure if it's completely solvable though, because any kind of S3 listing modification is subject to eventual consistency effects. I'm not planning to change anything related to this within the scope of the current patch though.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 7m 4s trunk passed
          +1 compile 0m 11s trunk passed with JDK v1.8.0_91
          +1 compile 0m 13s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 31s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 10s the patch passed with JDK v1.8.0_91
          +1 javac 0m 10s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.7.0_95
          +1 javac 0m 12s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 16s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 0m 38s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          13m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802848/HADOOP-13113.002.patch
          JIRA Issue HADOOP-13113
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ab94ac778226 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 47c41e7
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9328/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9328/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 4s trunk passed +1 compile 0m 11s trunk passed with JDK v1.8.0_91 +1 compile 0m 13s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 14s the patch passed +1 compile 0m 10s the patch passed with JDK v1.8.0_91 +1 javac 0m 10s the patch passed +1 compile 0m 12s the patch passed with JDK v1.7.0_95 +1 javac 0m 12s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_95 +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 13m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802848/HADOOP-13113.002.patch JIRA Issue HADOOP-13113 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ab94ac778226 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 47c41e7 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9328/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9328/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching patch v002, adding TestJets3tNativeS3FileSystemContract to the include/exclude lists. This is in anticipation that someday we'll resolve HADOOP-13116 too. It's another test suite that can't run in parallel with others.

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching patch v002, adding TestJets3tNativeS3FileSystemContract to the include/exclude lists. This is in anticipation that someday we'll resolve HADOOP-13116 too. It's another test suite that can't run in parallel with others.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 6m 53s trunk passed
          +1 compile 0m 12s trunk passed with JDK v1.8.0_91
          +1 compile 0m 13s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 18s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_91
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 13s the patch passed
          +1 compile 0m 10s the patch passed with JDK v1.8.0_91
          +1 javac 0m 10s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.7.0_95
          +1 javac 0m 11s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 16s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 0m 38s the patch passed
          +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
          +1 unit 0m 10s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          13m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802806/HADOOP-13113.001.patch
          JIRA Issue HADOOP-13113
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 6fbbab01fa6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 47c41e7
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9321/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9321/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 6m 53s trunk passed +1 compile 0m 12s trunk passed with JDK v1.8.0_91 +1 compile 0m 13s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_91 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 13s the patch passed +1 compile 0m 10s the patch passed with JDK v1.8.0_91 +1 javac 0m 10s the patch passed +1 compile 0m 11s the patch passed with JDK v1.7.0_95 +1 javac 0m 11s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 0m 10s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 13m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802806/HADOOP-13113.001.patch JIRA Issue HADOOP-13113 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 6fbbab01fa6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 47c41e7 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9321/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9321/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I'm attaching patch v001. This uses the same parallel test execution techniques that have been effective in hadoop-common and hadoop-hdfs. I updated pom.xml to add the parallel-tests profile and changed some test code to generate testing paths based on the Surefire fork number, so that they're guaranteed to be isolated from each other. HADOOP-12801 is a pre-requisite for this patch.

          There are some limitations to this approach. Tests that cover root directory access, multi-part upload, or YARN usage would run into conflicts on shared resources. To handle this, I've moved those tests to a separate Surefire execution step, which runs sequentially instead of parallelized. A future improvement could be to refactor tests related to root out of FileSystemContractBaseTest, so that the non-root tests in that suite can run in parallel. TestS3AFileSystemContract takes ~90 seconds to run, so it would be nice to parallelize that with other suites. I chose not to step outside hadoop-aws for the scope of the current patch though.

          While I was editing pom.xml, I also removed an unused dependency on Jackson. This change is backwards-compatible, because clients who had been relying on receiving Jackson via transitive dependency through hadoop-aws will continue to receive it through the dependency chain hadoop-aws -> hadoop-common -> Jackson.

          In my environment, plain mvn test runs in ~30 minutes, and mvn -Pparallel-tests test runs in ~13 minutes.

          Steve Loughran, would you please review when you get a chance?

          Show
          cnauroth Chris Nauroth added a comment - I'm attaching patch v001. This uses the same parallel test execution techniques that have been effective in hadoop-common and hadoop-hdfs. I updated pom.xml to add the parallel-tests profile and changed some test code to generate testing paths based on the Surefire fork number, so that they're guaranteed to be isolated from each other. HADOOP-12801 is a pre-requisite for this patch. There are some limitations to this approach. Tests that cover root directory access, multi-part upload, or YARN usage would run into conflicts on shared resources. To handle this, I've moved those tests to a separate Surefire execution step, which runs sequentially instead of parallelized. A future improvement could be to refactor tests related to root out of FileSystemContractBaseTest , so that the non-root tests in that suite can run in parallel. TestS3AFileSystemContract takes ~90 seconds to run, so it would be nice to parallelize that with other suites. I chose not to step outside hadoop-aws for the scope of the current patch though. While I was editing pom.xml, I also removed an unused dependency on Jackson. This change is backwards-compatible, because clients who had been relying on receiving Jackson via transitive dependency through hadoop-aws will continue to receive it through the dependency chain hadoop-aws -> hadoop-common -> Jackson. In my environment, plain mvn test runs in ~30 minutes, and mvn -Pparallel-tests test runs in ~13 minutes. Steve Loughran , would you please review when you get a chance?

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development