Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13208

S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      S3A has optimized the listFiles method by doing a bulk listing of all entries under a path in a single S3 operation instead of recursively walking the directory tree. The listLocatedStatus method has been optimized by fetching results from S3 lazily as the caller traverses the returned iterator instead of doing an eager fetch of all possible results.
      Show
      S3A has optimized the listFiles method by doing a bulk listing of all entries under a path in a single S3 operation instead of recursively walking the directory tree. The listLocatedStatus method has been optimized by fetching results from S3 lazily as the caller traverses the returned iterator instead of doing an eager fetch of all possible results.

      Description

      A major cost in split calculation against object stores turns out be listing the directory tree itself. That's because against S3, it takes S3A two HEADs and two lists to list the content of any directory path (2 HEADs + 1 list for getFileStatus(); the next list to query the contents).

      Listing a directory could be improved slightly by combining the final two listings. However, a listing of a directory tree will still be O(directories). In contrast, a recursive listFiles() operation should be implementable by a bulk listing of all descendant paths; one List operation per thousand descendants.

      As the result of this call is an iterator, the ongoing listing can be implemented within the iterator itself

      1. HADOOP-13208-branch-2-021.patch
        50 kB
        Chris Nauroth
      2. HADOOP-13208-branch-2-020.patch
        50 kB
        Steve Loughran
      3. HADOOP-13208-branch-2-019.patch
        43 kB
        Steve Loughran
      4. HADOOP-13208-branch-2-018.patch
        42 kB
        Steve Loughran
      5. HADOOP-13208-branch-2-017.patch
        165 kB
        Steve Loughran
      6. HADOOP-13208-branch-2-012.patch
        113 kB
        Steve Loughran
      7. HADOOP-13208-branch-2-011.patch
        112 kB
        Steve Loughran
      8. HADOOP-13208-branch-2-010.patch
        105 kB
        Steve Loughran
      9. HADOOP-13208-branch-2-009.patch
        105 kB
        Steve Loughran
      10. HADOOP-13208-branch-2-008.patch
        90 kB
        Steve Loughran
      11. HADOOP-13208-branch-2-007.patch
        90 kB
        Steve Loughran
      12. HADOOP-13208-branch-2-001.patch
        89 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment - - edited

          I cherry-picked this to branch-2.8 for inclusion in 2.8.0.

          Show
          cnauroth Chris Nauroth added a comment - - edited I cherry-picked this to branch-2.8 for inclusion in 2.8.0.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10294 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10294/)
          HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects (cnauroth: rev 822d661b8fcc42bec6eea958d9fd02ef1aaa4b6c)

          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/TestS3AContractGetFileStatus.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
          • (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3ADirectoryPerformance.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10294 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10294/ ) HADOOP-13208 . S3A listFiles(recursive=true) to do a bulk listObjects (cnauroth: rev 822d661b8fcc42bec6eea958d9fd02ef1aaa4b6c) (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/TestS3AContractGetFileStatus.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3ADirectoryPerformance.java
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch 021. I have committed this to trunk and branch-2. Steve Loughran, thank you for the contribution. Aaron Fabbri, thank you for your code review.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch 021. I have committed this to trunk and branch-2. Steve Loughran , thank you for the contribution. Aaron Fabbri , thank you for your code review.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 6m 27s branch-2 passed
          +1 compile 0m 13s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 15s branch-2 passed
          +1 mvnsite 0m 21s branch-2 passed
          +1 mvneclipse 0m 15s branch-2 passed
          +1 findbugs 0m 33s branch-2 passed
          +1 javadoc 0m 13s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_101
          +1 javac 0m 11s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_101
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 0 new + 27 unchanged - 2 fixed = 27 total (was 29)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_101
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          13m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824238/HADOOP-13208-branch-2-021.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux add1b9855dba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 2486c4c
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10285/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10285/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 6m 27s branch-2 passed +1 compile 0m 13s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 15s branch-2 passed +1 mvnsite 0m 21s branch-2 passed +1 mvneclipse 0m 15s branch-2 passed +1 findbugs 0m 33s branch-2 passed +1 javadoc 0m 13s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 14s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_101 +1 javac 0m 11s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_101 +1 javac 0m 13s the patch passed +1 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 0 new + 27 unchanged - 2 fixed = 27 total (was 29) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_101 +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 13m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824238/HADOOP-13208-branch-2-021.patch JIRA Issue HADOOP-13208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux add1b9855dba 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 2486c4c Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10285/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10285/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve, thank you for patch 020. I verified all tests passing when applied to trunk, running against an S3 bucket in US-west-2. Refactoring to the new Listing class is helpful.

          I'm attaching patch 021 with some trivial changes to get us a clean pre-commit run:

          • Correct JavaDoc parameter name that was causing a JavaDoc failure for Java 8.
          • Wrap some lines that were longer than 80 characters in Listing.
          Show
          cnauroth Chris Nauroth added a comment - Steve, thank you for patch 020. I verified all tests passing when applied to trunk, running against an S3 bucket in US-west-2. Refactoring to the new Listing class is helpful. I'm attaching patch 021 with some trivial changes to get us a clean pre-commit run: Correct JavaDoc parameter name that was causing a JavaDoc failure for Java 8. Wrap some lines that were longer than 80 characters in Listing .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 6m 33s branch-2 passed
          +1 compile 0m 13s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 14s branch-2 passed
          +1 mvnsite 0m 21s branch-2 passed
          +1 mvneclipse 0m 14s branch-2 passed
          +1 findbugs 0m 34s branch-2 passed
          +1 javadoc 0m 13s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_101
          +1 javac 0m 11s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_101
          +1 javac 0m 13s the patch passed
          -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 2 new + 27 unchanged - 2 fixed = 29 total (was 29)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 44s the patch passed
          -1 javadoc 0m 11s hadoop-aws in the patch failed with JDK v1.8.0_101.
          -1 javadoc 0m 13s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          14m 6s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824216/HADOOP-13208-branch-2-020.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b20884244812 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 848ae35
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws-jdk1.8.0_101.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 6m 33s branch-2 passed +1 compile 0m 13s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 15s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 14s branch-2 passed +1 mvnsite 0m 21s branch-2 passed +1 mvneclipse 0m 14s branch-2 passed +1 findbugs 0m 34s branch-2 passed +1 javadoc 0m 13s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 14s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_101 +1 javac 0m 11s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_101 +1 javac 0m 13s the patch passed -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 2 new + 27 unchanged - 2 fixed = 29 total (was 29) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed -1 javadoc 0m 11s hadoop-aws in the patch failed with JDK v1.8.0_101. -1 javadoc 0m 13s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 14m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824216/HADOOP-13208-branch-2-020.patch JIRA Issue HADOOP-13208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b20884244812 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 848ae35 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws-jdk1.8.0_101.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10283/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 020

          • moves out all the new listing classes into a new class Listing, which is bonded to an S3a owner and makes calls on it to do the listing.
          • tries to use package-private to reduce scope of member access changes and class visibility.
          • addresses chris's javadoc comment

          regarding the trunk merge; looks like theres an out-of-order import in trunk that 's been fixed in Branch-2; the new imports for listings hit this. Moving the code out will dodge the problem, though it still lurks

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 020 moves out all the new listing classes into a new class Listing , which is bonded to an S3a owner and makes calls on it to do the listing. tries to use package-private to reduce scope of member access changes and class visibility. addresses chris's javadoc comment regarding the trunk merge; looks like theres an out-of-order import in trunk that 's been fixed in Branch-2; the new imports for listings hit this. Moving the code out will dodge the problem, though it still lurks
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I will look at trunk. I've just been pulling out all the Listing classes into their own package-private class, to keep S3AFileSystem slightly more under control

          Show
          stevel@apache.org Steve Loughran added a comment - I will look at trunk. I've just been pulling out all the Listing classes into their own package-private class, to keep S3AFileSystem slightly more under control
          Hide
          cnauroth Chris Nauroth added a comment -
             * 1. The first invocation of the {@link #next()} call will return the results
             * of the first request, the one passed in during the construction of the
             * instance.
          

          This comment is no longer accurate after the move of the listing call to within the constructor.

          Checkstyle flagged an unused import warning. It wasn't introduced by this patch, but if you feel like cleaning it up, please go ahead.

          Otherwise, patch 019 looks good. I'll wait until resolution of the trunk problem before I do my final review and full test pass. It looks like just a minor conflict on some import statements, so hopefully it's easy to create a trunk-compatible patch.

          Show
          cnauroth Chris Nauroth added a comment - * 1. The first invocation of the {@link #next()} call will return the results * of the first request, the one passed in during the construction of the * instance. This comment is no longer accurate after the move of the listing call to within the constructor. Checkstyle flagged an unused import warning. It wasn't introduced by this patch, but if you feel like cleaning it up, please go ahead. Otherwise, patch 019 looks good. I'll wait until resolution of the trunk problem before I do my final review and full test pass. It looks like just a minor conflict on some import statements, so hopefully it's easy to create a trunk-compatible patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          The patch does not apply to trunk. I haven't investigated why.

          Show
          cnauroth Chris Nauroth added a comment - The patch does not apply to trunk. I haven't investigated why.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 11m 11s branch-2 passed
          +1 compile 0m 15s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 16s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 18s branch-2 passed
          +1 mvnsite 0m 25s branch-2 passed
          +1 mvneclipse 1m 8s branch-2 passed
          +1 findbugs 0m 41s branch-2 passed
          +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_101
          +1 javac 0m 12s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_101
          +1 javac 0m 14s the patch passed
          -0 checkstyle 0m 14s hadoop-tools/hadoop-aws: The patch generated 1 new + 28 unchanged - 1 fixed = 29 total (was 29)
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 49s the patch passed
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_101
          +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          20m 31s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824189/HADOOP-13208-branch-2-019.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1b4a37b8e2f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 848ae35
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10282/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10282/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10282/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 11m 11s branch-2 passed +1 compile 0m 15s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 16s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 18s branch-2 passed +1 mvnsite 0m 25s branch-2 passed +1 mvneclipse 1m 8s branch-2 passed +1 findbugs 0m 41s branch-2 passed +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 16s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 15s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_101 +1 javac 0m 12s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_101 +1 javac 0m 14s the patch passed -0 checkstyle 0m 14s hadoop-tools/hadoop-aws: The patch generated 1 new + 28 unchanged - 1 fixed = 29 total (was 29) +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 49s the patch passed +1 javadoc 0m 13s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_101 +1 unit 0m 15s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 20m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824189/HADOOP-13208-branch-2-019.patch JIRA Issue HADOOP-13208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1b4a37b8e2f3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 848ae35 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10282/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10282/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10282/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 019

          @Aaron

          Returning the array directly; also renamed the param 'f' to 'path' for clarity.

          @cnauroth

          Why split the two iterators up? to make it suspiciously close to java-8-lang lambdas and (ii) hopefully to line things up better for a high performance globber.

          2. I've pushed the list call down into the `ObjectListingIterator` but left delimiter calculation to the caller.

          3. FileStatusGenerator. I had a plan there to maybe generate liststatus values directly, but I'll postpone that plan & have inlined thaat code.

          nits: all addressed

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 019 @Aaron Returning the array directly; also renamed the param 'f' to 'path' for clarity. @cnauroth Why split the two iterators up? to make it suspiciously close to java-8-lang lambdas and (ii) hopefully to line things up better for a high performance globber. 2. I've pushed the list call down into the `ObjectListingIterator` but left delimiter calculation to the caller. 3. FileStatusGenerator. I had a plan there to maybe generate liststatus values directly, but I'll postpone that plan & have inlined thaat code. nits: all addressed
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve Loughran, thank you for the patch. I'm still working through some of the logic in detail, but this looks great overall. I have a few questions and comments so far.

          The control flow across the various iterators is a bit hard to follow. I think this is an unavoidable consequence of the optimized logic, but I'd like to suggest some things that might make it easier to follow.

          1. I notice that all call sites first contruct an ObjectListingIterator and then immediately pass it into construction of a FileStatusListingIterator. Since the two classes cannot be used independently in a meaningful way, I wonder if it would be better to combine all of the logic into a single class. If not combined, then perhaps it would be helpful to have a createFileStatusListingIterator helper method to encapsulate the 2-step construction, and all call sites could call that.
          2. Could the first listing call be pushed into the ObjectListingIterator constructor? That would have the benefit of the iterator fully encapsulating all of the calls so that each call site doesn't need to bootstrap it with the first listing call. If the constructor also accepted a recursive flag, then it could take care of deciding whether or not to pass a "/" delimiter: another detail that call sites wouldn't need to worry about getting right.
          3. It appears that any reference to a FileStatusGenerator is always going to point to an instance of a single implementation: GenerateS3AFileStatus. Is there a reason for the extra indirection, or can the logic of GenerateS3AFileStatus be inlined to the call sites? (Maybe the indirection sets up something helpful in a subsequent patch that I haven't reviewed yet?)
          4. I could see moving the new inner classes to top-level package-private classes, assuming that wouldn't force relaxing visibility on S3AFileSystem internals too much.
             * This implementation is optimized for S3, which can do a bulk listing
             * off all entries under a path in one single operation. Thus there is
             * no need to recursively walk the directory tree.
          

          This comment on listLocatedStatus confused me, because this method does not recursively traverse the sub-tree. As I understand it, this patch does not reduce the number of S3 calls during a listLocatedStatus operation, assuming that the caller fully exhausts the returned iterator. However, it's still a good change, because if the caller breaks out of the iteration early, then they don't pay the full cost of an eager listStatus fetch (multiple S3 calls) that the base class FileSystem#listLocatedStatus would do.

          Do I understand correctly, or did I miss something recursive about this operation?

          A few nitpicks:

             * @return the fully qualified path including URI schema and bucket name.
          

          Should be "URI scheme"?

             * Make this protected method public so that {@link S3AGlobber can access it}.
          ...
             * Override superclass to use the new {@code S3AGlobber}.
          

          Since the globber stuff is going to be done in a different patch, I suggest omitting these changes from this patch.

             * This essentially a nested and wrapped set of iterators, with some
          

          Should be "is essentially"?

              builder.append("size=").append(summary.getSize()).append(" ");
              return builder.toString();
          

          The last append of a trailing space looks unnecessary.

                result = new ArrayList<>(1);
          

          I agree with Aaron's earlier comment about returning an array directly here and only allocating an ArrayList on the is-directory code path.

          Show
          cnauroth Chris Nauroth added a comment - Steve Loughran , thank you for the patch. I'm still working through some of the logic in detail, but this looks great overall. I have a few questions and comments so far. The control flow across the various iterators is a bit hard to follow. I think this is an unavoidable consequence of the optimized logic, but I'd like to suggest some things that might make it easier to follow. I notice that all call sites first contruct an ObjectListingIterator and then immediately pass it into construction of a FileStatusListingIterator . Since the two classes cannot be used independently in a meaningful way, I wonder if it would be better to combine all of the logic into a single class. If not combined, then perhaps it would be helpful to have a createFileStatusListingIterator helper method to encapsulate the 2-step construction, and all call sites could call that. Could the first listing call be pushed into the ObjectListingIterator constructor? That would have the benefit of the iterator fully encapsulating all of the calls so that each call site doesn't need to bootstrap it with the first listing call. If the constructor also accepted a recursive flag, then it could take care of deciding whether or not to pass a "/" delimiter: another detail that call sites wouldn't need to worry about getting right. It appears that any reference to a FileStatusGenerator is always going to point to an instance of a single implementation: GenerateS3AFileStatus . Is there a reason for the extra indirection, or can the logic of GenerateS3AFileStatus be inlined to the call sites? (Maybe the indirection sets up something helpful in a subsequent patch that I haven't reviewed yet?) I could see moving the new inner classes to top-level package-private classes, assuming that wouldn't force relaxing visibility on S3AFileSystem internals too much. * This implementation is optimized for S3, which can do a bulk listing * off all entries under a path in one single operation. Thus there is * no need to recursively walk the directory tree. This comment on listLocatedStatus confused me, because this method does not recursively traverse the sub-tree. As I understand it, this patch does not reduce the number of S3 calls during a listLocatedStatus operation, assuming that the caller fully exhausts the returned iterator. However, it's still a good change, because if the caller breaks out of the iteration early, then they don't pay the full cost of an eager listStatus fetch (multiple S3 calls) that the base class FileSystem#listLocatedStatus would do. Do I understand correctly, or did I miss something recursive about this operation? A few nitpicks: * @ return the fully qualified path including URI schema and bucket name. Should be "URI scheme"? * Make this protected method public so that {@link S3AGlobber can access it}. ... * Override superclass to use the new {@code S3AGlobber}. Since the globber stuff is going to be done in a different patch, I suggest omitting these changes from this patch. * This essentially a nested and wrapped set of iterators, with some Should be "is essentially"? builder.append( "size=" ).append(summary.getSize()).append( " " ); return builder.toString(); The last append of a trailing space looks unnecessary. result = new ArrayList<>(1); I agree with Aaron's earlier comment about returning an array directly here and only allocating an ArrayList on the is-directory code path.
          Hide
          cnauroth Chris Nauroth added a comment -

          What is this for? I see it was introduced in HADOOP-13131, but don't see
          any usage of the configuration flag it sets (fs.s3a.impl.disable.cache).

          Aaron Fabbri, sorry, I missed this comment last time around.

          This logic disables the FileSystem cache. The logic for this is inside the FileSystem base class. (See the static FileSystem#get(URI, Configuration) method.) Whether or not the cache is in effect is determined by a dynamic configuration property: fs.<scheme>.impl.disable.cache.

          In some cases, we might have a test suite with multiple test methods that all want to test slightly different configuration. However, all tests in a suite execute in the same process, so there would be a risk of reusing a cached instance across multiple tests. Unfortunately, the caching is not sensitive to changes in the Configuration instance. (The cache key is the scheme, authority, and UserGroupInformation.) To work around that, we disable the cache for these tests.

          Show
          cnauroth Chris Nauroth added a comment - What is this for? I see it was introduced in HADOOP-13131 , but don't see any usage of the configuration flag it sets (fs.s3a.impl.disable.cache). Aaron Fabbri , sorry, I missed this comment last time around. This logic disables the FileSystem cache. The logic for this is inside the FileSystem base class. (See the static FileSystem#get(URI, Configuration) method.) Whether or not the cache is in effect is determined by a dynamic configuration property: fs.<scheme>.impl.disable.cache . In some cases, we might have a test suite with multiple test methods that all want to test slightly different configuration. However, all tests in a suite execute in the same process, so there would be a risk of reusing a cached instance across multiple tests. Unfortunately, the caching is not sensitive to changes in the Configuration instance. (The cache key is the scheme, authority, and UserGroupInformation .) To work around that, we disable the cache for these tests.
          Hide
          cnauroth Chris Nauroth added a comment -

          Aaron Fabbri, thank you for your code review. This is near the head of my review queue, so I'll look soon too. Steve is out for several weeks, so it will take a while before he can respond to code review feedback.

          I feel like we need to break up S3AFileSystem.java some.

          Yes, I agree, and I think this is a requirement for S3Guard. I have a breakdown in mind that I plan to propose as a sub-task of HADOOP-13345.

          Show
          cnauroth Chris Nauroth added a comment - Aaron Fabbri , thank you for your code review. This is near the head of my review queue, so I'll look soon too. Steve is out for several weeks, so it will take a while before he can respond to code review feedback. I feel like we need to break up S3AFileSystem.java some. Yes, I agree, and I think this is a requirement for S3Guard. I have a breakdown in mind that I plan to propose as a sub-task of HADOOP-13345 .
          Hide
          fabbri Aaron Fabbri added a comment -

          I spent a good amount of time this evening going over patch #18. Overall looks good, considering the previous JIRA included most of the test code. Are there any unit tests still missing (I'm willing to help if so)?

          I'm +1 (non-binding).

          Some random comments / questions:

          +
               } else {
                 LOG.debug("Adding: rd (not a dir): {}", f);
          +      result = new ArrayList<>(1);
                 result.add(fileStatus);
          

          How about just returning the singleton array here, to reduce garbage a bit?

          +  private ListObjectsRequest createListObjectsRequest(String key,
          +      String delimiter) {
          +    ListObjectsRequest request = new ListObjectsRequest();
          +    request.setBucketName(bucket);
          +    request.setMaxKeys(maxKeys);
          +    request.setPrefix(key);
          +    if (delimiter != null) {
          +      request.setDelimiter(delimiter);
          +    }
          +    return request;
          +  }
          

          Good factorization.

          +  private final class FileStatusListingIterator
          +      implements RemoteIterator<FileStatus> {
          +
          +    /** Source of objects. */
          +    private final ObjectListingIterator source;
          

          I feel like we need to break up S3AFileSystem.java some. It is getting pretty
          large. We should take any easy opportunities to pull out classes into separate
          files. I'm fine with doing this as a follow-up JIRA.

          +     */
          +    private boolean requestNextBatch() throws IOException {
          +      // look for more object listing batches being available
          +      while (source.hasNext()) {
          +        // if available, retrieve it and build the next status
          +        if (buildNextStatusBatch(source.next())) {
          

          Yeah, this logic is non-trivial. This while loop is key.

          -   */
          -  static class AWSAccessKeys {
          -    private String accessKey = null;
          -    private String accessSecret = null;
          

          Bonus deadc0de removal, ok.

          +  @Override
          +  protected Configuration createConfiguration() {
          +    Configuration conf = super.createConfiguration();
          +    S3ATestUtils.disableFilesystemCaching(conf);
          

          What is this for? I see it was introduced in HADOOP-13131, but don't see
          any usage of the configuration flag it sets (fs.s3a.impl.disable.cache).

          Show
          fabbri Aaron Fabbri added a comment - I spent a good amount of time this evening going over patch #18. Overall looks good, considering the previous JIRA included most of the test code. Are there any unit tests still missing (I'm willing to help if so)? I'm +1 (non-binding). Some random comments / questions: + } else { LOG.debug( "Adding: rd (not a dir): {}" , f); + result = new ArrayList<>(1); result.add(fileStatus); How about just returning the singleton array here, to reduce garbage a bit? + private ListObjectsRequest createListObjectsRequest( String key, + String delimiter) { + ListObjectsRequest request = new ListObjectsRequest(); + request.setBucketName(bucket); + request.setMaxKeys(maxKeys); + request.setPrefix(key); + if (delimiter != null ) { + request.setDelimiter(delimiter); + } + return request; + } Good factorization. + private final class FileStatusListingIterator + implements RemoteIterator<FileStatus> { + + /** Source of objects. */ + private final ObjectListingIterator source; I feel like we need to break up S3AFileSystem.java some. It is getting pretty large. We should take any easy opportunities to pull out classes into separate files. I'm fine with doing this as a follow-up JIRA. + */ + private boolean requestNextBatch() throws IOException { + // look for more object listing batches being available + while (source.hasNext()) { + // if available, retrieve it and build the next status + if (buildNextStatusBatch(source.next())) { Yeah, this logic is non-trivial. This while loop is key. - */ - static class AWSAccessKeys { - private String accessKey = null ; - private String accessSecret = null ; Bonus deadc0de removal, ok. + @Override + protected Configuration createConfiguration() { + Configuration conf = super .createConfiguration(); + S3ATestUtils.disableFilesystemCaching(conf); What is this for? I see it was introduced in HADOOP-13131 , but don't see any usage of the configuration flag it sets (fs.s3a.impl.disable.cache).
          Hide
          fabbri Aaron Fabbri added a comment -

          Abdullah Yousufi listStatus() already retrieves file info in batches, and it is not recursive. This patch is specific to a recursive list operation: It uses the ability of s3 to list by key (path) prefix to optimize round trips to accomplish the equivalent of ls -R.

          Show
          fabbri Aaron Fabbri added a comment - Abdullah Yousufi listStatus() already retrieves file info in batches, and it is not recursive. This patch is specific to a recursive list operation: It uses the ability of s3 to list by key (path) prefix to optimize round trips to accomplish the equivalent of ls -R .
          Hide
          ayousufi Abdullah Yousufi added a comment -

          Hey Steve Loughran, is there a reason this change only applies to listFiles() and not listStatus()?

          Show
          ayousufi Abdullah Yousufi added a comment - Hey Steve Loughran , is there a reason this change only applies to listFiles() and not listStatus()?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 10m 45s branch-2 passed
          +1 compile 0m 14s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 16s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 17s branch-2 passed
          +1 mvnsite 0m 22s branch-2 passed
          +1 mvneclipse 1m 29s branch-2 passed
          +1 findbugs 0m 39s branch-2 passed
          +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_101
          +1 javac 0m 11s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_101
          +1 javac 0m 13s the patch passed
          -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 1 new + 28 unchanged - 1 fixed = 29 total (was 29)
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 42s the patch passed
          -1 javadoc 0m 10s hadoop-aws in the patch failed with JDK v1.8.0_101.
          -1 javadoc 0m 13s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          32m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819779/HADOOP-13208-branch-2-018.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 20a2edd40b09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / e6ba69b
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws-jdk1.8.0_101.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 10m 45s branch-2 passed +1 compile 0m 14s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 16s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 17s branch-2 passed +1 mvnsite 0m 22s branch-2 passed +1 mvneclipse 1m 29s branch-2 passed +1 findbugs 0m 39s branch-2 passed +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 14s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_101 +1 javac 0m 11s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_101 +1 javac 0m 13s the patch passed -0 checkstyle 0m 12s hadoop-tools/hadoop-aws: The patch generated 1 new + 28 unchanged - 1 fixed = 29 total (was 29) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 42s the patch passed -1 javadoc 0m 10s hadoop-aws in the patch failed with JDK v1.8.0_101. -1 javadoc 0m 13s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 32m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819779/HADOOP-13208-branch-2-018.patch JIRA Issue HADOOP-13208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 20a2edd40b09 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e6ba69b Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws-jdk1.8.0_101.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10068/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          tested against s3 ireland

          Show
          stevel@apache.org Steve Loughran added a comment - tested against s3 ireland
          Hide
          stevel@apache.org Steve Loughran added a comment -

          The no of requests is 5+(N/5000) for the curious, where N= # of files + mock directories under path P.

          Contrast with the recursive walker, where time is something like 4* real-or-mock-directories + files/5000

          Show
          stevel@apache.org Steve Loughran added a comment - The no of requests is 5+(N/5000) for the curious, where N= # of files + mock directories under path P . Contrast with the recursive walker, where time is something like 4* real-or-mock-directories + files/5000
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 018; in sync with branch-2; all glob code stripped

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 018; in sync with branch-2; all glob code stripped
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 10 new or modified test files.
          0 mvndep 10m 59s Maven dependency ordering for branch
          +1 mvninstall 7m 1s branch-2 passed
          +1 compile 5m 50s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 28s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 25s branch-2 passed
          +1 mvnsite 1m 40s branch-2 passed
          +1 mvneclipse 1m 44s branch-2 passed
          +1 findbugs 2m 53s branch-2 passed
          +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 5m 39s the patch passed with JDK v1.8.0_91
          +1 javac 5m 39s the patch passed
          +1 compile 6m 28s the patch passed with JDK v1.7.0_101
          +1 javac 6m 28s the patch passed
          -0 checkstyle 1m 25s root: The patch generated 15 new + 62 unchanged - 52 fixed = 77 total (was 114)
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 3m 35s the patch passed
          +1 javadoc 1m 28s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 43s the patch passed with JDK v1.7.0_101
          +1 unit 7m 44s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 20s hadoop-openstack in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          107m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819725/HADOOP-13208-branch-2-017.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b0e5d669296c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 2f1e352
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 10 new or modified test files. 0 mvndep 10m 59s Maven dependency ordering for branch +1 mvninstall 7m 1s branch-2 passed +1 compile 5m 50s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 28s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 25s branch-2 passed +1 mvnsite 1m 40s branch-2 passed +1 mvneclipse 1m 44s branch-2 passed +1 findbugs 2m 53s branch-2 passed +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 5m 39s the patch passed with JDK v1.8.0_91 +1 javac 5m 39s the patch passed +1 compile 6m 28s the patch passed with JDK v1.7.0_101 +1 javac 6m 28s the patch passed -0 checkstyle 1m 25s root: The patch generated 15 new + 62 unchanged - 52 fixed = 77 total (was 114) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 53s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 35s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_101 +1 unit 7m 44s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 20s hadoop-openstack in the patch passed with JDK v1.7.0_101. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 107m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819725/HADOOP-13208-branch-2-017.patch JIRA Issue HADOOP-13208 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b0e5d669296c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 2f1e352 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10065/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Tested against S3 ireland; all well.

          Show
          stevel@apache.org Steve Loughran added a comment - Tested against S3 ireland; all well.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is the aws module in sync with HADOOP-13207 patch 017, for curious

          this diff also contains something which isn't ready for review (and which I will move off to the side): the tests and initial implementation of an s3a-specific globber. Regression tests are all off the HDFS glob tests.

          There's no attempt at optimising the s3a globber yet; that will be the real work for the globber patch and I haven't yet started that.

          Show
          stevel@apache.org Steve Loughran added a comment - This is the aws module in sync with HADOOP-13207 patch 017, for curious this diff also contains something which isn't ready for review (and which I will move off to the side): the tests and initial implementation of an s3a-specific globber. Regression tests are all off the HDFS glob tests. There's no attempt at optimising the s3a globber yet; that will be the real work for the globber patch and I haven't yet started that.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 7m 42s branch-2 passed
          +1 compile 8m 1s branch-2 passed with JDK v1.8.0_91
          +1 compile 7m 46s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 30s branch-2 passed
          +1 mvnsite 1m 52s branch-2 passed
          +1 mvneclipse 0m 48s branch-2 passed
          +1 findbugs 3m 12s branch-2 passed
          +1 javadoc 1m 32s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 37s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 22s the patch passed
          +1 compile 7m 10s the patch passed with JDK v1.8.0_91
          +1 javac 7m 10s the patch passed
          +1 compile 6m 35s the patch passed with JDK v1.7.0_101
          +1 javac 6m 35s the patch passed
          +1 checkstyle 1m 24s root: The patch generated 0 new + 58 unchanged - 52 fixed = 58 total (was 110)
          +1 mvnsite 1m 46s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 3m 47s the patch passed
          +1 javadoc 1m 38s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 49s the patch passed with JDK v1.7.0_101
          -1 unit 8m 6s hadoop-common in the patch failed with JDK v1.7.0_101.
          +1 unit 0m 19s hadoop-openstack in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          105m 37s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching
            hadoop.metrics2.impl.TestGangliaMetrics
          JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817732/HADOOP-13208-branch-2-012.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 3cf9ce5ebc45 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 694b4ab
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 42s branch-2 passed +1 compile 8m 1s branch-2 passed with JDK v1.8.0_91 +1 compile 7m 46s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 30s branch-2 passed +1 mvnsite 1m 52s branch-2 passed +1 mvneclipse 0m 48s branch-2 passed +1 findbugs 3m 12s branch-2 passed +1 javadoc 1m 32s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 37s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 7m 10s the patch passed with JDK v1.8.0_91 +1 javac 7m 10s the patch passed +1 compile 6m 35s the patch passed with JDK v1.7.0_101 +1 javac 6m 35s the patch passed +1 checkstyle 1m 24s root: The patch generated 0 new + 58 unchanged - 52 fixed = 58 total (was 110) +1 mvnsite 1m 46s the patch passed +1 mvneclipse 0m 53s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 47s the patch passed +1 javadoc 1m 38s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 49s the patch passed with JDK v1.7.0_101 -1 unit 8m 6s hadoop-common in the patch failed with JDK v1.7.0_101. +1 unit 0m 19s hadoop-openstack in the patch passed with JDK v1.7.0_101. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 105m 37s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching   hadoop.metrics2.impl.TestGangliaMetrics JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817732/HADOOP-13208-branch-2-012.patch JIRA Issue HADOOP-13208 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 3cf9ce5ebc45 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 694b4ab Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9988/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-13208 patch 012: javadoc failing now that some inner classes were private; converted @link to @code

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-13208 patch 012: javadoc failing now that some inner classes were private; converted @link to @code
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 1m 6s Maven dependency ordering for branch
          +1 mvninstall 6m 55s branch-2 passed
          +1 compile 6m 11s branch-2 passed with JDK v1.8.0_91
          +1 compile 7m 11s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 24s branch-2 passed
          +1 mvnsite 1m 43s branch-2 passed
          +1 mvneclipse 0m 45s branch-2 passed
          +1 findbugs 3m 0s branch-2 passed
          +1 javadoc 1m 21s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 35s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 22s the patch passed
          +1 compile 6m 29s the patch passed with JDK v1.8.0_91
          +1 javac 6m 29s the patch passed
          +1 compile 6m 44s the patch passed with JDK v1.7.0_101
          +1 javac 6m 44s the patch passed
          +1 checkstyle 1m 24s root: The patch generated 0 new + 58 unchanged - 52 fixed = 58 total (was 110)
          +1 mvnsite 1m 47s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 3m 32s the patch passed
          +1 javadoc 1m 28s the patch passed with JDK v1.8.0_91
          -1 javadoc 0m 21s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          -1 unit 7m 57s hadoop-common in the patch failed with JDK v1.7.0_101.
          +1 unit 0m 17s hadoop-openstack in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          109m 4s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching
          JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817680/HADOOP-13208-branch-2-011.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux d32aa85d907d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 694b4ab
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 1m 6s Maven dependency ordering for branch +1 mvninstall 6m 55s branch-2 passed +1 compile 6m 11s branch-2 passed with JDK v1.8.0_91 +1 compile 7m 11s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 24s branch-2 passed +1 mvnsite 1m 43s branch-2 passed +1 mvneclipse 0m 45s branch-2 passed +1 findbugs 3m 0s branch-2 passed +1 javadoc 1m 21s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 35s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 6m 29s the patch passed with JDK v1.8.0_91 +1 javac 6m 29s the patch passed +1 compile 6m 44s the patch passed with JDK v1.7.0_101 +1 javac 6m 44s the patch passed +1 checkstyle 1m 24s root: The patch generated 0 new + 58 unchanged - 52 fixed = 58 total (was 110) +1 mvnsite 1m 47s the patch passed +1 mvneclipse 0m 53s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 32s the patch passed +1 javadoc 1m 28s the patch passed with JDK v1.8.0_91 -1 javadoc 0m 21s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) -1 unit 7m 57s hadoop-common in the patch failed with JDK v1.7.0_101. +1 unit 0m 17s hadoop-openstack in the patch passed with JDK v1.7.0_101. +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 109m 4s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching JDK v1.8.0_91 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817680/HADOOP-13208-branch-2-011.patch JIRA Issue HADOOP-13208 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux d32aa85d907d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 694b4ab Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9987/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          tested agains s3 ireland. With the modified test, I'm showing a 20x speedup for listing the specific directory tree created for this test:

          2016-07-13 14:56:42,370 [Thread-8] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1362)) - Duration of List status via treewalk of 48 directories and 6 files: 5,964,773,672 nS
          2016-07-13 14:56:42,370 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_metadata_requests starting=444 current=542 diff=98
          2016-07-13 14:56:42,370 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_list_requests starting=132 current=184 diff=52
          2016-07-13 14:56:42,370 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_continue_list_requests starting=0 current=0 diff=0
          2016-07-13 14:56:42,370 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_list_files starting=0 current=0 diff=0
          2016-07-13 14:56:42,370 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_get_file_status starting=225 current=274 diff=49
          2016-07-13 14:56:42,371 [Thread-8] INFO  scale.S3AScaleTestBase (S3AScaleTestBase.java:describe(155)) - 
          
          testListOperations: Listing files via listFiles(recursive=true)
          
          2016-07-13 14:56:42,579 [Thread-8] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1362)) - Duration of listFiles(recursive=true) of 48 directories and 6 files: 208,461,316 nS
          2016-07-13 14:56:42,580 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_metadata_requests starting=542 current=544 diff=2
          2016-07-13 14:56:42,580 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_list_requests starting=184 current=186 diff=2
          2016-07-13 14:56:42,580 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_continue_list_requests starting=0 current=0 diff=0
          2016-07-13 14:56:42,580 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_list_files starting=0 current=1 diff=1
          2016-07-13 14:56:42,580 [Thread-8] INFO  scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_get_file_status starting=274 current=275 diff=1
          2016-07-13 14:56:42,580 [Thread-8] INFO  scale.S3AScaleTestBase (S3AScaleTestBase.java:describe(155)) - 
          

          Obviously, that could be un-representative, but it does show the speedup we can obtain

          Show
          stevel@apache.org Steve Loughran added a comment - tested agains s3 ireland. With the modified test, I'm showing a 20x speedup for listing the specific directory tree created for this test: 2016-07-13 14:56:42,370 [ Thread -8] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1362)) - Duration of List status via treewalk of 48 directories and 6 files: 5,964,773,672 nS 2016-07-13 14:56:42,370 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_metadata_requests starting=444 current=542 diff=98 2016-07-13 14:56:42,370 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_list_requests starting=132 current=184 diff=52 2016-07-13 14:56:42,370 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_continue_list_requests starting=0 current=0 diff=0 2016-07-13 14:56:42,370 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_list_files starting=0 current=0 diff=0 2016-07-13 14:56:42,370 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_get_file_status starting=225 current=274 diff=49 2016-07-13 14:56:42,371 [ Thread -8] INFO scale.S3AScaleTestBase (S3AScaleTestBase.java:describe(155)) - testListOperations: Listing files via listFiles(recursive= true ) 2016-07-13 14:56:42,579 [ Thread -8] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1362)) - Duration of listFiles(recursive= true ) of 48 directories and 6 files: 208,461,316 nS 2016-07-13 14:56:42,580 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_metadata_requests starting=542 current=544 diff=2 2016-07-13 14:56:42,580 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_list_requests starting=184 current=186 diff=2 2016-07-13 14:56:42,580 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - object_continue_list_requests starting=0 current=0 diff=0 2016-07-13 14:56:42,580 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_list_files starting=0 current=1 diff=1 2016-07-13 14:56:42,580 [ Thread -8] INFO scale.TestS3ADirectoryPerformance (S3ATestUtils.java:print(212)) - op_get_file_status starting=274 current=275 diff=1 2016-07-13 14:56:42,580 [ Thread -8] INFO scale.S3AScaleTestBase (S3AScaleTestBase.java:describe(155)) - Obviously, that could be un-representative, but it does show the speedup we can obtain
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 011: fix up TestS3ADirectoryPerformance to correctly validate the number of requests made during the listStatus() call, changed the assert to only expect two object list requests (the initial is-directory probe and the followup listing). Split the counter of object list requests into one for the initial request and one for followup actions.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 011: fix up TestS3ADirectoryPerformance to correctly validate the number of requests made during the listStatus() call, changed the assert to only expect two object list requests (the initial is-directory probe and the followup listing). Split the counter of object list requests into one for the initial request and one for followup actions.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          build failed because sourceforge is playing up

          Show
          stevel@apache.org Steve Loughran added a comment - build failed because sourceforge is playing up
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 5m 46s Docker failed to build yetus/hadoop:b59b8b7.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817451/HADOOP-13208-branch-2-009.patch
          JIRA Issue HADOOP-13208
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9981/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 5m 46s Docker failed to build yetus/hadoop:b59b8b7. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817451/HADOOP-13208-branch-2-009.patch JIRA Issue HADOOP-13208 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9981/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 010. Address checkstyle and findbugs warnings.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 010. Address checkstyle and findbugs warnings.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 8m 56s Maven dependency ordering for branch
          +1 mvninstall 6m 59s branch-2 passed
          +1 compile 5m 34s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 29s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 26s branch-2 passed
          +1 mvnsite 1m 42s branch-2 passed
          +1 mvneclipse 3m 12s branch-2 passed
          +1 findbugs 2m 53s branch-2 passed
          +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 5m 27s the patch passed with JDK v1.8.0_91
          +1 javac 5m 27s the patch passed
          +1 compile 6m 33s the patch passed with JDK v1.7.0_101
          +1 javac 6m 33s the patch passed
          -0 checkstyle 1m 25s root: The patch generated 7 new + 40 unchanged - 52 fixed = 47 total (was 92)
          +1 mvnsite 1m 47s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 0m 52s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 1m 35s the patch passed with JDK v1.8.0_91
          -1 javadoc 0m 22s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 unit 9m 12s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          110m 20s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Dead store to result in org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:[line 1227]
            Should org.apache.hadoop.fs.s3a.S3AFileSystem$SingleFileRemoteIterator be a static inner class? At S3AFileSystem.java:inner class? At S3AFileSystem.java:[lines 2155-2199]
          JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817451/HADOOP-13208-branch-2-009.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 14fc1397099b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / e94e6be
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 8m 56s Maven dependency ordering for branch +1 mvninstall 6m 59s branch-2 passed +1 compile 5m 34s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 29s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 26s branch-2 passed +1 mvnsite 1m 42s branch-2 passed +1 mvneclipse 3m 12s branch-2 passed +1 findbugs 2m 53s branch-2 passed +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 5m 27s the patch passed with JDK v1.8.0_91 +1 javac 5m 27s the patch passed +1 compile 6m 33s the patch passed with JDK v1.7.0_101 +1 javac 6m 33s the patch passed -0 checkstyle 1m 25s root: The patch generated 7 new + 40 unchanged - 52 fixed = 47 total (was 92) +1 mvnsite 1m 47s the patch passed +1 mvneclipse 0m 53s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 52s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 1m 35s the patch passed with JDK v1.8.0_91 -1 javadoc 0m 22s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 unit 9m 12s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_101. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 110m 20s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Dead store to result in org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java: [line 1227]   Should org.apache.hadoop.fs.s3a.S3AFileSystem$SingleFileRemoteIterator be a static inner class? At S3AFileSystem.java:inner class? At S3AFileSystem.java: [lines 2155-2199] JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817451/HADOOP-13208-branch-2-009.patch JIRA Issue HADOOP-13208 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 14fc1397099b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e94e6be Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9967/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 009.

          1. specify RemoteIterator in filesystem.md, specifically that implementations MUST return finite sequence, that a while(true) next(); is a valid iteration, and cover concurrency issues.
          2. listing tests also grab the remote iterators returned and do the while(true) evaluation, verifying the results match.
          3. Fix S3AFileSystem iterators to correctly pass these new tests (the other filesystems, which don't override the default list* operations, all pass)
          4. javadoc all RemoteIterators in S3AFileSystem, so explaining chained operations
          Show
          stevel@apache.org Steve Loughran added a comment - Patch 009. specify RemoteIterator in filesystem.md , specifically that implementations MUST return finite sequence, that a while(true) next(); is a valid iteration, and cover concurrency issues. listing tests also grab the remote iterators returned and do the while(true) evaluation, verifying the results match. Fix S3AFileSystem iterators to correctly pass these new tests (the other filesystems, which don't override the default list* operations, all pass) javadoc all RemoteIterators in S3AFileSystem, so explaining chained operations
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 008; rebased to branch-2...some merge problems related to import declarations in S3AFileSystem fixed

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 008; rebased to branch-2...some merge problems related to import declarations in S3AFileSystem fixed
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 39s Maven dependency ordering for branch
          +1 mvninstall 6m 50s branch-2 passed
          +1 compile 7m 25s branch-2 passed with JDK v1.8.0_91
          +1 compile 7m 44s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 36s branch-2 passed
          +1 mvnsite 1m 54s branch-2 passed
          +1 mvneclipse 0m 48s branch-2 passed
          +1 findbugs 3m 13s branch-2 passed
          +1 javadoc 1m 28s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 38s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 7m 16s the patch passed with JDK v1.8.0_91
          +1 javac 7m 16s the patch passed
          +1 compile 7m 16s the patch passed with JDK v1.7.0_101
          +1 javac 7m 16s the patch passed
          -0 checkstyle 1m 31s root: The patch generated 9 new + 42 unchanged - 52 fixed = 51 total (was 94)
          +1 mvnsite 2m 3s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          -1 whitespace 0m 1s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 0m 59s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 1m 24s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 41s the patch passed with JDK v1.7.0_101
          +1 unit 9m 16s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 23s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          107m 51s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Dead store to result in org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:[line 1280]
            Should org.apache.hadoop.fs.s3a.S3AFileSystem$SingleFileRemoteIterator be a static inner class? At S3AFileSystem.java:inner class? At S3AFileSystem.java:[lines 2065-2089]
          JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching
            hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12815475/HADOOP-13208-branch-2-007.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 865ce93aa5a9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 1458e66
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 39s Maven dependency ordering for branch +1 mvninstall 6m 50s branch-2 passed +1 compile 7m 25s branch-2 passed with JDK v1.8.0_91 +1 compile 7m 44s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 36s branch-2 passed +1 mvnsite 1m 54s branch-2 passed +1 mvneclipse 0m 48s branch-2 passed +1 findbugs 3m 13s branch-2 passed +1 javadoc 1m 28s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 38s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 7m 16s the patch passed with JDK v1.8.0_91 +1 javac 7m 16s the patch passed +1 compile 7m 16s the patch passed with JDK v1.7.0_101 +1 javac 7m 16s the patch passed -0 checkstyle 1m 31s root: The patch generated 9 new + 42 unchanged - 52 fixed = 51 total (was 94) +1 mvnsite 2m 3s the patch passed +1 mvneclipse 0m 55s the patch passed -1 whitespace 0m 1s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 59s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 1m 24s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_101 +1 unit 9m 16s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_101. +1 unit 0m 23s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 107m 51s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Dead store to result in org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java: [line 1280]   Should org.apache.hadoop.fs.s3a.S3AFileSystem$SingleFileRemoteIterator be a static inner class? At S3AFileSystem.java:inner class? At S3AFileSystem.java: [lines 2065-2089] JDK v1.8.0_91 Failed junit tests hadoop.security.TestGroupsCaching   hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12815475/HADOOP-13208-branch-2-007.patch JIRA Issue HADOOP-13208 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 865ce93aa5a9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 1458e66 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9913/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is patch -007, which I'm numbering to keep in sync with the HADOOP-13207 patch of the same version.

          This patch has the filesystem.md doc changes of that patch, and fixes the javadoc merge error that had crept in during rebasing.

          Test: S3 Ireland

          Show
          stevel@apache.org Steve Loughran added a comment - This is patch -007, which I'm numbering to keep in sync with the HADOOP-13207 patch of the same version. This patch has the filesystem.md doc changes of that patch, and fixes the javadoc merge error that had crept in during rebasing. Test: S3 Ireland
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I've been doing the two together in the source tree, but yes, I should be splitting them out just by doing the diff of hadoop-common standalone. I'll do that for the ext iteration.

          Show
          stevel@apache.org Steve Loughran added a comment - I've been doing the two together in the source tree, but yes, I should be splitting them out just by doing the diff of hadoop-common standalone. I'll do that for the ext iteration.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          that's why I missed it then. Will fix

          Show
          stevel@apache.org Steve Loughran added a comment - that's why I missed it then. Will fix
          Hide
          fabbri Aaron Fabbri added a comment -

          Should we get HADOOP-13207 committed and then post a new patch here with just the listFiles() changes?

          Show
          fabbri Aaron Fabbri added a comment - Should we get HADOOP-13207 committed and then post a new patch here with just the listFiles() changes?
          Hide
          fabbri Aaron Fabbri added a comment - - edited

          Looks like there is a merge conflict in S3AFileSystem.java:

          +<<<<<<< 3907f2a30d773828e85cb3c7ea367be749b65920
              * Check that a Path belongs to this FileSystem.
              * Unlike the superclass, this version does not look at authority,
              * only hostnames.
              * @param path to check
              * @throws IllegalArgumentException if there is an FS mismatch
          +=======
          +   * Convert a key to a fully qualified path.
          +   * @param key input key
          +   * @return the fully qualified path including URI schema and bucket name.
          +   */
          +  private Path keyToQualifiedPath(String key) {
          +    return keyToPath(key).makeQualified(uri, workingDir);
          +  }
          +
          +  /**
          +   * Opens an FSDataInputStream at the indicated Path.
          +   * @param f the file name to open
          +   * @param bufferSize the size of the buffer to be used.
          +>>>>>>> HADOOP-13208: build remote iterator direct from object listings; use in the nonrecursive listLocatedStatus call
          

          Funny part is that the markers land in comments so it still compiles.

          Show
          fabbri Aaron Fabbri added a comment - - edited Looks like there is a merge conflict in S3AFileSystem.java: +<<<<<<< 3907f2a30d773828e85cb3c7ea367be749b65920 * Check that a Path belongs to this FileSystem. * Unlike the superclass, this version does not look at authority, * only hostnames. * @param path to check * @ throws IllegalArgumentException if there is an FS mismatch +======= + * Convert a key to a fully qualified path. + * @param key input key + * @ return the fully qualified path including URI schema and bucket name. + */ + private Path keyToQualifiedPath( String key) { + return keyToPath(key).makeQualified(uri, workingDir); + } + + /** + * Opens an FSDataInputStream at the indicated Path. + * @param f the file name to open + * @param bufferSize the size of the buffer to be used. +>>>>>>> HADOOP-13208: build remote iterator direct from object listings; use in the nonrecursive listLocatedStatus call Funny part is that the markers land in comments so it still compiles.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 58s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 37s branch-2 passed
          +1 compile 7m 18s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 53s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 24s branch-2 passed
          +1 mvnsite 1m 41s branch-2 passed
          +1 mvneclipse 0m 44s branch-2 passed
          +1 findbugs 2m 50s branch-2 passed
          +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 5m 57s the patch passed with JDK v1.8.0_91
          +1 javac 5m 57s the patch passed
          +1 compile 7m 1s the patch passed with JDK v1.7.0_101
          +1 javac 7m 1s the patch passed
          -0 checkstyle 1m 23s root: The patch generated 10 new + 42 unchanged - 52 fixed = 52 total (was 94)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 0m 53s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          -1 javadoc 0m 18s hadoop-aws in the patch failed with JDK v1.8.0_91.
          -1 javadoc 0m 21s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 unit 7m 52s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 23s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          99m 39s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Dead store to result in org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:[line 1286]
            Should org.apache.hadoop.fs.s3a.S3AFileSystem$SingleFileRemoteIterator be a static inner class? At S3AFileSystem.java:inner class? At S3AFileSystem.java:[lines 2071-2095]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814796/HADOOP-13208-branch-2-001.patch
          JIRA Issue HADOOP-13208
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 6d570d6016c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 1285035
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws-jdk1.8.0_91.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 58s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 37s branch-2 passed +1 compile 7m 18s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 53s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 24s branch-2 passed +1 mvnsite 1m 41s branch-2 passed +1 mvneclipse 0m 44s branch-2 passed +1 findbugs 2m 50s branch-2 passed +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 5m 57s the patch passed with JDK v1.8.0_91 +1 javac 5m 57s the patch passed +1 compile 7m 1s the patch passed with JDK v1.7.0_101 +1 javac 7m 1s the patch passed -0 checkstyle 1m 23s root: The patch generated 10 new + 42 unchanged - 52 fixed = 52 total (was 94) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 49s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 53s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) -1 javadoc 0m 18s hadoop-aws in the patch failed with JDK v1.8.0_91. -1 javadoc 0m 21s hadoop-tools_hadoop-aws-jdk1.7.0_101 with JDK v1.7.0_101 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 unit 7m 52s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 21s hadoop-openstack in the patch passed with JDK v1.7.0_101. +1 unit 0m 23s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 99m 39s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Dead store to result in org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java:org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(Path) At S3AFileSystem.java: [line 1286]   Should org.apache.hadoop.fs.s3a.S3AFileSystem$SingleFileRemoteIterator be a static inner class? At S3AFileSystem.java:inner class? At S3AFileSystem.java: [lines 2071-2095] Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12814796/HADOOP-13208-branch-2-001.patch JIRA Issue HADOOP-13208 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 6d570d6016c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 1285035 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9903/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001

          incorporates HADOOP-13207

          • define what the various list* commands do
          • adds tests in the get file status contract test suite, which are automatically picked up by all implementation subclasses classes.
          • adds some root dir listing tests to AbstractContractRootDirectoryTest, not because they manipulate the root, but to guarantee that test runs which parallelize the tests (e.g. hadoop-aws) run the root listing tests in a suite which would have to be serialized already.

          And in S3A

          • pull out the listObject request construction
          • add a RemoteIterator ObjectListingIterator to go through the results of a listing
          • add the RemoteIterator implementations FileStatusListingIterator and LocatedFileStatusIterator to take an ObjectListingIterator, do filtering and file status creation when iterated over.
          • changed the listing operations to use these iterators, where appropriate.

          The result is that we have a fairly functional programming-ish model of working across the listed entities, and can do what is essentially O(1) listing of directory trees. Specifically, It's 1+ O(files/max-keys): the number of child directories is not a factor except in returning some extra paths to be discarded.

          Testing: S3 frankfurt, also tested azure, openstack, localfs, hdfs

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001 incorporates HADOOP-13207 define what the various list* commands do adds tests in the get file status contract test suite, which are automatically picked up by all implementation subclasses classes. adds some root dir listing tests to AbstractContractRootDirectoryTest , not because they manipulate the root, but to guarantee that test runs which parallelize the tests (e.g. hadoop-aws) run the root listing tests in a suite which would have to be serialized already. And in S3A pull out the listObject request construction add a RemoteIterator ObjectListingIterator to go through the results of a listing add the RemoteIterator implementations FileStatusListingIterator and LocatedFileStatusIterator to take an ObjectListingIterator , do filtering and file status creation when iterated over. changed the listing operations to use these iterators, where appropriate. The result is that we have a fairly functional programming-ish model of working across the listed entities, and can do what is essentially O(1) listing of directory trees. Specifically, It's 1+ O(files/max-keys) : the number of child directories is not a factor except in returning some extra paths to be discarded. Testing: S3 frankfurt, also tested azure, openstack, localfs, hdfs
          Hide
          stevel@apache.org Steve Loughran added a comment -

          One little detail, nothing has been written down about how listFiles() really works, there are no FS contract tests. HADOOP-13207 covers this.

          Show
          stevel@apache.org Steve Loughran added a comment - One little detail, nothing has been written down about how listFiles() really works, there are no FS contract tests. HADOOP-13207 covers this.

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development