Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10784

Implement WebHdfsFileSystem#listStatusIterator

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: webhdfs
    • Labels:
      None
    • Target Version/s:

      Description

      It would be nice to implement the iterative listStatus in WebHDFS so client apps do not need to buffer the full file list for large directories.

      1. HDFS-10784.001.patch
        18 kB
        Andrew Wang
      2. HDFS-10784.002.patch
        20 kB
        Andrew Wang
      3. HDFS-10784.003.patch
        27 kB
        Andrew Wang

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Patch attached. I couldn't find a nice way of codesharing with the other iterator implementations, but it isn't that much.

          Show
          andrew.wang Andrew Wang added a comment - Patch attached. I couldn't find a nice way of codesharing with the other iterator implementations, but it isn't that much.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 30s Maven dependency ordering for branch
          +1 mvninstall 7m 24s trunk passed
          +1 compile 1m 28s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 7s trunk passed
          +1 javadoc 1m 16s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 21s the patch passed
          -1 javac 1m 21s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52)
          -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 10 new + 252 unchanged - 3 fixed = 262 total (was 255)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 44s the patch passed
          +1 javadoc 1m 17s the patch passed
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
          -1 unit 78m 58s hadoop-hdfs in the patch failed.
          -1 asflicense 0m 22s The patch generated 1 ASF License warnings.
          108m 25s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.mover.TestStorageMover



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824969/HDFS-10784.001.patch
          JIRA Issue HDFS-10784
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 24f0506a7f26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c49333b
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16503/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16503/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 7m 24s trunk passed +1 compile 1m 28s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 7s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 21s the patch passed -1 javac 1m 21s hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52 total (was 52) -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 10 new + 252 unchanged - 3 fixed = 262 total (was 255) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 44s the patch passed +1 javadoc 1m 17s the patch passed +1 unit 1m 2s hadoop-hdfs-client in the patch passed. -1 unit 78m 58s hadoop-hdfs in the patch failed. -1 asflicense 0m 22s The patch generated 1 ASF License warnings. 108m 25s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.mover.TestStorageMover Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824969/HDFS-10784.001.patch JIRA Issue HDFS-10784 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 24f0506a7f26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c49333b Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16503/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16503/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Looks like similar to HDFS-9366..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Looks like similar to HDFS-9366 ..?
          Hide
          andrew.wang Andrew Wang added a comment -

          Good find Brahma, somehow I forgot about that one. Yea, it does look pretty similar, though HDFS-9366 doesn't implement the RemoteIterator interface. I'll look at HDFS-9366 to see if I missed anything else though.

          Show
          andrew.wang Andrew Wang added a comment - Good find Brahma, somehow I forgot about that one. Yea, it does look pretty similar, though HDFS-9366 doesn't implement the RemoteIterator interface. I'll look at HDFS-9366 to see if I missed anything else though.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew for working on this, and Brahma for bringing up HDFS-9366. Patch itself looks good.

          The feature the 2 jiras want to solve looks very alike - allowing some pagination when listing. Implementation is a little different:

          • HDFS-10784 implements RemoteIterator, so adds a new interface listStatusIterator.
          • HDFS-9366 overloads current listStatus, with customized offset and size parameters.

          IMHO, HDFS-10784 is cleaner and more flexible, hence easier to use when user wants to iterate the whole listing. HDFS-9366 could have less end-to-end trips when listing with a starting offset not at the beginning.
          Looking at DistributedFileSystem, an iterator would be more consistent with hdfs context. I don't have a strong opinion, maybe we should ask our user Romain Rigaux since both jira seems to aim at Hue.

          I think we should combine this and HDFS-9366 after agreement, and add documentation. Would be great if httpfs is supported too.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew for working on this, and Brahma for bringing up HDFS-9366 . Patch itself looks good. The feature the 2 jiras want to solve looks very alike - allowing some pagination when listing. Implementation is a little different: HDFS-10784 implements RemoteIterator , so adds a new interface listStatusIterator . HDFS-9366 overloads current listStatus , with customized offset and size parameters. IMHO, HDFS-10784 is cleaner and more flexible, hence easier to use when user wants to iterate the whole listing. HDFS-9366 could have less end-to-end trips when listing with a starting offset not at the beginning. Looking at DistributedFileSystem , an iterator would be more consistent with hdfs context. I don't have a strong opinion, maybe we should ask our user Romain Rigaux since both jira seems to aim at Hue. I think we should combine this and HDFS-9366 after agreement, and add documentation. Would be great if httpfs is supported too.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for taking a look Xiao. FWIW, the limitation of starting the listing at an offset is a function of the Java API, but not the REST API itself. Since Hue doesn't use the Java API, it can implement an "iterator at offset" if it so desires. We could implement it in the Java API too if that's desired, but I'm hesitant to widen the FileSystem API any further.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for taking a look Xiao. FWIW, the limitation of starting the listing at an offset is a function of the Java API, but not the REST API itself. Since Hue doesn't use the Java API, it can implement an "iterator at offset" if it so desires. We could implement it in the Java API too if that's desired, but I'm hesitant to widen the FileSystem API any further.
          Hide
          andrew.wang Andrew Wang added a comment -

          I added some more tests and tried to fix the precommit errors, let's see what Mr. Jenkins says.

          Show
          andrew.wang Andrew Wang added a comment - I added some more tests and tried to fix the precommit errors, let's see what Mr. Jenkins says.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 28s Maven dependency ordering for branch
          +1 mvninstall 8m 52s trunk passed
          +1 compile 1m 51s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 52s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 48s trunk passed
          +1 javadoc 1m 14s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 27s the patch passed
          +1 javac 1m 27s the patch passed
          -0 checkstyle 0m 34s hadoop-hdfs-project: The patch generated 4 new + 252 unchanged - 3 fixed = 256 total (was 255)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 24s the patch passed
          +1 javadoc 1m 15s the patch passed
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          -1 unit 89m 23s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          121m 38s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10784
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825748/HDFS-10784.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fd76beb391ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 19c743c
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16555/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16555/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16555/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 28s Maven dependency ordering for branch +1 mvninstall 8m 52s trunk passed +1 compile 1m 51s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 48s trunk passed +1 javadoc 1m 14s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 1m 27s the patch passed +1 javac 1m 27s the patch passed -0 checkstyle 0m 34s hadoop-hdfs-project: The patch generated 4 new + 252 unchanged - 3 fixed = 256 total (was 255) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 1m 15s the patch passed +1 unit 0m 56s hadoop-hdfs-client in the patch passed. -1 unit 89m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 121m 38s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10784 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825748/HDFS-10784.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fd76beb391ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19c743c Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16555/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16555/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16555/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Checkstyles are because of existing coding style, test failures look unrelated. Review appreciated!

          Show
          andrew.wang Andrew Wang added a comment - Checkstyles are because of existing coding style, test failures look unrelated. Review appreciated!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the new patch, Andrew. Looks pretty good to me.
          I have a question, and some nits.

          Question:

          • I see httpfs' documentation says HttpFs... And it is inteoperable with the webhdfs REST HTTP API. Does this mean we have to implement this for httpfs too, in order to keep this interoperability?

          Nits (in nitness order, the later the nitter...):

          • I think we should add this new API to WebHDFS.md with an example.
          • Maybe we should also test the iterator on empty dir and / or dir with 1 entry.
          • Extra line break at the beginning of JsonUtilClient#toDirectoryListing and StartAfterParam class declaration.
          • WebHdfsFileSystem: there're already com.google imports in the later part of the class, let's put them together.
          Show
          xiaochen Xiao Chen added a comment - Thanks for the new patch, Andrew. Looks pretty good to me. I have a question, and some nits. Question: I see httpfs' documentation says HttpFs... And it is inteoperable with the webhdfs REST HTTP API. Does this mean we have to implement this for httpfs too, in order to keep this interoperability? Nits (in nitness order, the later the nitter...): I think we should add this new API to WebHDFS.md with an example. Maybe we should also test the iterator on empty dir and / or dir with 1 entry. Extra line break at the beginning of JsonUtilClient#toDirectoryListing and StartAfterParam class declaration. WebHdfsFileSystem : there're already com.google imports in the later part of the class, let's put them together.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for reviewing Xiao. Yep, we should hit HttpFS also, but if you don't mind, I'd like to do it in a separate JIRA. Feedback is otherwise addressed.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for reviewing Xiao. Yep, we should hit HttpFS also, but if you don't mind, I'd like to do it in a separate JIRA. Feedback is otherwise addressed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 27s Maven dependency ordering for branch
          +1 mvninstall 7m 23s trunk passed
          +1 compile 1m 30s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 3m 12s trunk passed
          +1 javadoc 1m 15s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 24s the patch passed
          +1 compile 1m 30s the patch passed
          +1 javac 1m 30s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 4 new + 252 unchanged - 3 fixed = 256 total (was 255)
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 3m 41s the patch passed
          +1 javadoc 1m 13s the patch passed
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          +1 unit 63m 41s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          93m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10784
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826210/HDFS-10784.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d0bb0ac586ee 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / af50860
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16578/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16578/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16578/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16578/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 27s Maven dependency ordering for branch +1 mvninstall 7m 23s trunk passed +1 compile 1m 30s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 3m 12s trunk passed +1 javadoc 1m 15s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 24s the patch passed +1 compile 1m 30s the patch passed +1 javac 1m 30s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 4 new + 252 unchanged - 3 fixed = 256 total (was 255) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 22s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 41s the patch passed +1 javadoc 1m 13s the patch passed +1 unit 0m 56s hadoop-hdfs-client in the patch passed. +1 unit 63m 41s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 93m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10784 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826210/HDFS-10784.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d0bb0ac586ee 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / af50860 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16578/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16578/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16578/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16578/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the new patch, Andrew.

          Checkstyle is not related, +1 pending the whitespace change.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the new patch, Andrew. Checkstyle is not related, +1 pending the whitespace change.
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk and branch-2, thanks for the review Xiao!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk and branch-2, thanks for the review Xiao!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10380 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10380/)
          HDFS-10784. Implement WebHdfsFileSystem#listStatusIterator. (wang: rev 85bab5fb572194fda38854f1f21c670925058009)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/StartAfterParam.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10380 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10380/ ) HDFS-10784 . Implement WebHdfsFileSystem#listStatusIterator. (wang: rev 85bab5fb572194fda38854f1f21c670925058009) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/StartAfterParam.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          Hide
          romainr Romain Rigaux added a comment -

          Thanks! And for HttpFs, should we create a follow-up jira?

          Show
          romainr Romain Rigaux added a comment - Thanks! And for HttpFs, should we create a follow-up jira?
          Hide
          xiaochen Xiao Chen added a comment -

          Yep, that's being tracked by the linked HDFS-10823.

          Show
          xiaochen Xiao Chen added a comment - Yep, that's being tracked by the linked HDFS-10823 .

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development