Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10837

Standardize serializiation of WebHDFS DirectoryListing

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: webhdfs
    • Labels:
      None

      Description

      HDFS-10784 introduced a batched listing API to WebHDFS. However, the API response doesn't follow the format of other WebHDFS calls. Let's standardize it, and also document the schema.

      1. hdfs-10837.001.patch
        16 kB
        Andrew Wang
      2. hdfs-10837.002.patch
        16 kB
        Andrew Wang
      3. hdfs-10837.003.patch
        16 kB
        Andrew Wang

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Patch attached. The key differences:

          • Define and use a named DirectoryListing map
          • Use a FileStatuses object for the value of the "partialListing" field
          • Add a schema definition for DirectoryListing
          Show
          andrew.wang Andrew Wang added a comment - Patch attached. The key differences: Define and use a named DirectoryListing map Use a FileStatuses object for the value of the "partialListing" field Add a schema definition for DirectoryListing
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 9m 11s trunk passed
          +1 compile 1m 28s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 13s trunk passed
          +1 javadoc 1m 23s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 1m 25s the patch passed
          +1 javac 1m 25s the patch passed
          -0 checkstyle 0m 27s hadoop-hdfs-project: The patch generated 1 new + 77 unchanged - 0 fixed = 78 total (was 77)
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 24s the patch passed
          +1 javadoc 1m 13s the patch passed
          +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
          -1 unit 73m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          103m 58s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10837
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827492/hdfs-10837.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0340629861b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d355573
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16669/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16669/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16669/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16669/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 9m 11s trunk passed +1 compile 1m 28s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 13s trunk passed +1 javadoc 1m 23s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 1m 25s the patch passed +1 javac 1m 25s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project: The patch generated 1 new + 77 unchanged - 0 fixed = 78 total (was 77) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 1m 13s the patch passed +1 unit 0m 58s hadoop-hdfs-client in the patch passed. -1 unit 73m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 103m 58s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10837 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827492/hdfs-10837.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0340629861b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d355573 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16669/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16669/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16669/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16669/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Checkstyle fixed, failed tests are a port conflict and HDFS-10499. Passed locally.

          Show
          andrew.wang Andrew Wang added a comment - Checkstyle fixed, failed tests are a port conflict and HDFS-10499 . Passed locally.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 8m 15s trunk passed
          +1 compile 1m 29s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 12s trunk passed
          +1 javadoc 1m 14s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 26s the patch passed
          +1 javac 1m 26s the patch passed
          +1 checkstyle 0m 32s the patch passed
          +1 mvnsite 1m 31s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 43s the patch passed
          +1 javadoc 1m 22s the patch passed
          +1 unit 1m 7s hadoop-hdfs-client in the patch passed.
          -1 unit 60m 51s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          91m 40s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.hdfs.TestMaintenanceState



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10837
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827845/hdfs-10837.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8f02cbad2f82 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bee9f57
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16700/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16700/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 8m 15s trunk passed +1 compile 1m 29s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 12s trunk passed +1 javadoc 1m 14s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 1m 26s the patch passed +1 javac 1m 26s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 43s the patch passed +1 javadoc 1m 22s the patch passed +1 unit 1m 7s hadoop-hdfs-client in the patch passed. -1 unit 60m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 91m 40s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.TestMaintenanceState Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10837 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827845/hdfs-10837.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8f02cbad2f82 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bee9f57 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16700/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16700/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew for catching and fixing this!

          Some small comments:

          • Let's add a input nullity check in toHdfsFileStatusArray
          • Maybe change assert array != null; to Precondition checks to be more consistent?
          • In the doc, the Iteratively_list_a_Directory link didn't work on my local. Iteratively_List_a_Directory worked.

          Test failures look unrelated. +1 after the above are addressed.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew for catching and fixing this! Some small comments: Let's add a input nullity check in toHdfsFileStatusArray Maybe change assert array != null; to Precondition checks to be more consistent? In the doc, the Iteratively_list_a_Directory link didn't work on my local. Iteratively_List_a_Directory worked. Test failures look unrelated. +1 after the above are addressed.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for reviewing Xiao, new patch attached. Added requested precondition checks and fixed the WebHDFS.md link (very thorough!).

          Show
          andrew.wang Andrew Wang added a comment - Thanks for reviewing Xiao, new patch attached. Added requested precondition checks and fixed the WebHDFS.md link (very thorough!).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 7m 29s trunk passed
          +1 compile 1m 39s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 48s trunk passed
          +1 javadoc 1m 33s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 35s the patch passed
          +1 compile 1m 28s the patch passed
          +1 javac 1m 28s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 27s the patch passed
          +1 javadoc 1m 13s the patch passed
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed.
          +1 unit 64m 54s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          95m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10837
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828148/hdfs-10837.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 49a82685a4d3 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 72dfb04
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16725/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16725/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 7m 29s trunk passed +1 compile 1m 39s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 48s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 35s the patch passed +1 compile 1m 28s the patch passed +1 javac 1m 28s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 27s the patch passed +1 javadoc 1m 13s the patch passed +1 unit 0m 55s hadoop-hdfs-client in the patch passed. +1 unit 64m 54s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 95m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10837 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828148/hdfs-10837.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 49a82685a4d3 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 72dfb04 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16725/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16725/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew for revving!
          The feature is covered on a wider test, so no tests needed for the smaller changes here. +1 on patch 3.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew for revving! The feature is covered on a wider test, so no tests needed for the smaller changes here. +1 on patch 3.
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk and branch-2, thanks for the reviews Xiao!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk and branch-2, thanks for the reviews Xiao!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10435 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10435/)
          HDFS-10837. Standardize serializiation of WebHDFS DirectoryListing. (wang: rev db6d243cf89d25fefbffd4c8721e14d9246b5a16)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10435 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10435/ ) HDFS-10837 . Standardize serializiation of WebHDFS DirectoryListing. (wang: rev db6d243cf89d25fefbffd4c8721e14d9246b5a16) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development