Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11156

Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: webhdfs
    • Labels:
      None

      Description

      Following webhdfs REST API

      http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GET_BLOCK_LOCATIONS&offset=0&length=1
      

      will get a response like

      {
        "LocatedBlocks" : {
          "fileLength" : 1073741824,
          "isLastBlockComplete" : true,
          "isUnderConstruction" : false,
          "lastLocatedBlock" : { ... },
          "locatedBlocks" : [ {...} ]
        }
      }
      

      This represents for o.a.h.h.p.LocatedBlocks. However according to FileSystem API,

      public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
      

      clients would expect an array of BlockLocation. This mismatch should be fixed. Marked as Incompatible change as this will change the output of the GET_BLOCK_LOCATIONS API.

      1. BlockLocationProperties_JSON_Schema.jpg
        283 kB
        Weiwei Yang
      2. BlockLocations_JSON_Schema.jpg
        167 kB
        Weiwei Yang
      3. FileStatuses_JSON_Schema.jpg
        147 kB
        Weiwei Yang
      4. HDFS-11156.01.patch
        15 kB
        Weiwei Yang
      5. HDFS-11156.02.patch
        16 kB
        Weiwei Yang
      6. HDFS-11156.03.patch
        16 kB
        Weiwei Yang
      7. HDFS-11156.04.patch
        15 kB
        Weiwei Yang
      8. HDFS-11156.05.patch
        11 kB
        Weiwei Yang
      9. HDFS-11156.06.patch
        12 kB
        Weiwei Yang
      10. HDFS-11156.07.patch
        20 kB
        Weiwei Yang
      11. HDFS-11156.08.patch
        29 kB
        Weiwei Yang
      12. HDFS-11156.09.patch
        29 kB
        Weiwei Yang
      13. HDFS-11156.10.patch
        30 kB
        Weiwei Yang
      14. HDFS-11156.11.patch
        30 kB
        Weiwei Yang
      15. HDFS-11156.12.patch
        30 kB
        Weiwei Yang
      16. HDFS-11156.13.patch
        31 kB
        Weiwei Yang
      17. HDFS-11156.14.patch
        31 kB
        Weiwei Yang
      18. HDFS-11156.15.patch
        33 kB
        Weiwei Yang
      19. HDFS-11156.16.patch
        33 kB
        Weiwei Yang
      20. HDFS-11156-branch-2.01.patch
        33 kB
        Weiwei Yang
      21. Output_JSON_format_v10.jpg
        220 kB
        Weiwei Yang
      22. SampleResponse_JSON.jpg
        248 kB
        Weiwei Yang

        Issue Links

          Activity

          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Andrew Wang, thank you, I have created HDFS-12459 to track the fix.

          Show
          cheersyang Weiwei Yang added a comment - Hi Andrew Wang , thank you, I have created HDFS-12459 to track the fix.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Weiwei, we need to leave this one resolved for changelog purposes since it was released in alpha2. Let's use a new JIRA to track the fixed work.

          Show
          andrew.wang Andrew Wang added a comment - Hi Weiwei, we need to leave this one resolved for changelog purposes since it was released in alpha2. Let's use a new JIRA to track the fixed work.
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks Rushabh S Shah, Daryn Sharp and Andrew Wang,

          Remove dfs client instantiation fromNamenodeWebHdfsMethods. Within NamenodeWebHdfsMethods, directly call NamenodeRpcServer related method which returns LocatedBlocks and convert it to BlockLocation[] inside NamenodeWebHdfsMethods

          This sounds good to me, I will submit a new patch based on the suggestions shortly.

          Show
          cheersyang Weiwei Yang added a comment - Thanks Rushabh S Shah , Daryn Sharp and Andrew Wang , Remove dfs client instantiation fromNamenodeWebHdfsMethods. Within NamenodeWebHdfsMethods, directly call NamenodeRpcServer related method which returns LocatedBlocks and convert it to BlockLocation[] inside NamenodeWebHdfsMethods This sounds good to me, I will submit a new patch based on the suggestions shortly.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12876 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12876/)
          Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST (wang: rev 390c2b5df0c4e25a2156d8daefd4670efe82b191)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12876 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12876/ ) Revert " HDFS-11156 . Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST (wang: rev 390c2b5df0c4e25a2156d8daefd4670efe82b191) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
          Hide
          shahrs87 Rushabh S Shah added a comment -

          What was the need for this jira ?

          Reading back all the comments in this jira, I understand what this jira is trying to fix.
          If we want to get this jira in 3-beta release, then I would suggest doing the following.
          Remove dfs client instantiation fromNamenodeWebHdfsMethods.
          Within NamenodeWebHdfsMethods,
          directly call NamenodeRpcServer related method which returns LocatedBlocks and convert it to BlockLocation[] inside NamenodeWebHdfsMethods.
          Remove all the code from WebHdfsFileSystem.
          Then I will remove my -1.

          Show
          shahrs87 Rushabh S Shah added a comment - What was the need for this jira ? Reading back all the comments in this jira, I understand what this jira is trying to fix. If we want to get this jira in 3-beta release, then I would suggest doing the following. Remove dfs client instantiation fromNamenodeWebHdfsMethods. Within NamenodeWebHdfsMethods , directly call NamenodeRpcServer related method which returns LocatedBlocks and convert it to BlockLocation[] inside NamenodeWebHdfsMethods . Remove all the code from WebHdfsFileSystem . Then I will remove my -1.
          Hide
          andrew.wang Andrew Wang added a comment -

          Reverted HDFS-6874 and HDFS-11156, using HDFS-12457 to track the revert from alpha2 to beta1.

          Show
          andrew.wang Andrew Wang added a comment - Reverted HDFS-6874 and HDFS-11156 , using HDFS-12457 to track the revert from alpha2 to beta1.
          Hide
          andrew.wang Andrew Wang added a comment -

          I'm going to revert this for now in the absence of agreement or a fix, since we're about to go to beta1.

          The usecase for this new call was to add a public API for REST-only cIients to get block locations. GET_BLOCK_LOCATIONS is an internal API (per Nicholas' comment above about the original design intent). Overall I'm ambivalent about this inclusion, since most people use the Java WebHDFS client, which calls the internal API.

          Show
          andrew.wang Andrew Wang added a comment - I'm going to revert this for now in the absence of agreement or a fix, since we're about to go to beta1. The usecase for this new call was to add a public API for REST-only cIients to get block locations. GET_BLOCK_LOCATIONS is an internal API (per Nicholas' comment above about the original design intent). Overall I'm ambivalent about this inclusion, since most people use the Java WebHDFS client, which calls the internal API.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          In addition to what Daryn Sharp said, we already had WebHdfsFileSystem#getFileBlockLocations added via HDFS-2340 back in 0.23 days.
          refer to 2.8 code since this patch didn't go to 2.8
          https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L1583

          clients would expect an array of BlockLocation.

          This api was returning array of BlockLocation. What was the need for this jira ?
          I am also +1 for reverting.

          Show
          shahrs87 Rushabh S Shah added a comment - In addition to what Daryn Sharp said, we already had WebHdfsFileSystem#getFileBlockLocations added via HDFS-2340 back in 0.23 days. refer to 2.8 code since this patch didn't go to 2.8 https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L1583 clients would expect an array of BlockLocation. This api was returning array of BlockLocation. What was the need for this jira ? I am also +1 for reverting.
          Hide
          daryn Daryn Sharp added a comment -

          Actually, why is the new webhdfs NN method even necessary? Why isn't the webhdfs method getFileBlockLocations just doing the conversion to BlockLocation[] just like the dfsclient does?

          Show
          daryn Daryn Sharp added a comment - Actually, why is the new webhdfs NN method even necessary? Why isn't the webhdfs method getFileBlockLocations just doing the conversion to BlockLocation[] just like the dfsclient does?
          Hide
          daryn Daryn Sharp added a comment -

          -1. Let's file another jira to fix immediately or revert this one. The NN cannot make RPC calls to itself. Other than completely unnecessary, it can cause all the jetty threads to be exhausted if they get into a retry loop with NN because it's too busy, in safemode, standby, etc. Notice that other webhdfs calls do not do this.

          There are no compatibility issues and no api needs to be changed. On the server-side, call the existing method and encode to same json format as today. On the client-side, just decode straight into a BlockLocation[].

          Show
          daryn Daryn Sharp added a comment - -1. Let's file another jira to fix immediately or revert this one. The NN cannot make RPC calls to itself. Other than completely unnecessary, it can cause all the jetty threads to be exhausted if they get into a retry loop with NN because it's too busy, in safemode, standby, etc. Notice that other webhdfs calls do not do this. There are no compatibility issues and no api needs to be changed. On the server-side, call the existing method and encode to same json format as today. On the client-side, just decode straight into a BlockLocation[].
          Hide
          shahrs87 Rushabh S Shah added a comment - - edited

          Current approach was trying to maintain compatibility as much as possible.

          I am unable to understand your point.
          DistributedFileSystem#getFileBlockLocations() after going through bunch of DFSClient related code calls ClientProtocol#getBlockLocations.
          NamenodeWebHdfsMethods#getRPCServer() returns NamenodeProtocols which in turn implements ClientProtocol also.
          So you can directly call np.getBlockLocations(src, offset, length) instead of fs.getFileBlockLocations(...) and then convert LocatedBlocks to BlockLocation[] using DFSUtilClient.locatedBlocks2Locations()
          Hope it makes sense.
          If not, I can write a small patch illustrating my idea.

          Show
          shahrs87 Rushabh S Shah added a comment - - edited Current approach was trying to maintain compatibility as much as possible. I am unable to understand your point. DistributedFileSystem#getFileBlockLocations() after going through bunch of DFSClient related code calls ClientProtocol#getBlockLocations . NamenodeWebHdfsMethods#getRPCServer() returns NamenodeProtocols which in turn implements ClientProtocol also. So you can directly call np.getBlockLocations(src, offset, length) instead of fs.getFileBlockLocations(...) and then convert LocatedBlocks to BlockLocation[] using DFSUtilClient.locatedBlocks2Locations() Hope it makes sense. If not, I can write a small patch illustrating my idea.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Rushabh S Shah

          Thanks for sharing your concern, I think you are raising up a good point. Current approach was trying to maintain compatibility as much as possible. We don't want to modify any public APIs because it will cause a lot of problems on upgrade/integrate paths. Ideally we could implement following API in ClientProtocol

          public BlockLocation[] getBlockLocations(String src,
            long start, long length) throws IOException;
          

          so that can be exposed via NamenodeRpcServer for NamenodeWebHdfsMethods to call. However because of the compatibility concerns I am not sure if this is worth to comparing to current path. It did a bit overhead on creating a dfs client, it is more compatible.

          Thanks

          Show
          cheersyang Weiwei Yang added a comment - Hi Rushabh S Shah Thanks for sharing your concern, I think you are raising up a good point. Current approach was trying to maintain compatibility as much as possible. We don't want to modify any public APIs because it will cause a lot of problems on upgrade/integrate paths. Ideally we could implement following API in ClientProtocol public BlockLocation[] getBlockLocations( String src, long start, long length) throws IOException; so that can be exposed via NamenodeRpcServer for NamenodeWebHdfsMethods to call. However because of the compatibility concerns I am not sure if this is worth to comparing to current path. It did a bit overhead on creating a dfs client, it is more compatible. Thanks
          Hide
          shahrs87 Rushabh S Shah added a comment -

          I have one concern in this patch.

          NamenodeWebHdfsMethods.java
          // Some comments here
          public String get()
          {
              case GETFILEBLOCKLOCATIONS:
              {
                final long offsetValue = offset.getValue();
                final Long lengthValue = length.getValue();
          
                FileSystem fs = FileSystem.get(conf != null ?
                    conf : new Configuration());
                BlockLocation[] locations = fs.getFileBlockLocations(
                    new org.apache.hadoop.fs.Path(fullpath),
                    offsetValue,
                    lengthValue != null? lengthValue: Long.MAX_VALUE);
                final String js = JsonUtil.toJsonString("BlockLocations",
                    JsonUtil.toJsonMap(locations));
                return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
              }
          }
          

          In the above code snippet, why are we creating dfsClient object which will again create an rpc to itself.
          We are already in namenode. We just have to call relevant NamenodeRpcServer method.
          Correct me if I am missing something.
          Cc Weiwei Yang Andrew Wang Mingliang Liu

          Also this fix introduced a bug. See more details here: HDFS-12442

          Show
          shahrs87 Rushabh S Shah added a comment - I have one concern in this patch. NamenodeWebHdfsMethods.java // Some comments here public String get() { case GETFILEBLOCKLOCATIONS: { final long offsetValue = offset.getValue(); final Long lengthValue = length.getValue(); FileSystem fs = FileSystem.get(conf != null ? conf : new Configuration()); BlockLocation[] locations = fs.getFileBlockLocations( new org.apache.hadoop.fs.Path(fullpath), offsetValue, lengthValue != null ? lengthValue: Long .MAX_VALUE); final String js = JsonUtil.toJsonString( "BlockLocations" , JsonUtil.toJsonMap(locations)); return Response.ok(js).type(MediaType.APPLICATION_JSON).build(); } } In the above code snippet, why are we creating dfsClient object which will again create an rpc to itself. We are already in namenode. We just have to call relevant NamenodeRpcServer method. Correct me if I am missing something. Cc Weiwei Yang Andrew Wang Mingliang Liu Also this fix introduced a bug. See more details here: HDFS-12442
          Hide
          andrew.wang Andrew Wang added a comment -

          I ran TestFailureToReadEdits locally and it passed, looks like a flake particularly since it worked on JDK8.

          I've committed this to branch-2, thanks again for the contribution Weiwei Yang!

          Show
          andrew.wang Andrew Wang added a comment - I ran TestFailureToReadEdits locally and it passed, looks like a flake particularly since it worked on JDK8. I've committed this to branch-2, thanks again for the contribution Weiwei Yang !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 9m 8s branch-2 passed
          +1 compile 1m 19s branch-2 passed with JDK v1.8.0_111
          +1 compile 1m 26s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 38s branch-2 passed
          +1 mvnsite 1m 31s branch-2 passed
          +1 mvneclipse 0m 30s branch-2 passed
          +1 findbugs 5m 12s branch-2 passed
          +1 javadoc 3m 8s branch-2 passed with JDK v1.8.0_111
          +1 javadoc 4m 13s branch-2 passed with JDK v1.7.0_121
          0 mvndep 0m 22s Maven dependency ordering for patch
          +1 mvninstall 2m 18s the patch passed
          +1 compile 2m 57s the patch passed with JDK v1.8.0_111
          +1 javac 2m 57s the patch passed
          +1 compile 1m 25s the patch passed with JDK v1.7.0_121
          +1 javac 1m 25s the patch passed
          -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 2 new + 242 unchanged - 1 fixed = 244 total (was 243)
          +1 mvnsite 1m 26s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 1s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_111
          +1 javadoc 1m 55s the patch passed with JDK v1.7.0_121
          +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121.
          -1 unit 51m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          169m 13s



          Reason Tests
          JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.namenode.TestStartup
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845470/HDFS-11156-branch-2.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux a77a6150771d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / db947fb
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18022/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18022/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18022/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18022/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 9m 8s branch-2 passed +1 compile 1m 19s branch-2 passed with JDK v1.8.0_111 +1 compile 1m 26s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 38s branch-2 passed +1 mvnsite 1m 31s branch-2 passed +1 mvneclipse 0m 30s branch-2 passed +1 findbugs 5m 12s branch-2 passed +1 javadoc 3m 8s branch-2 passed with JDK v1.8.0_111 +1 javadoc 4m 13s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 2m 18s the patch passed +1 compile 2m 57s the patch passed with JDK v1.8.0_111 +1 javac 2m 57s the patch passed +1 compile 1m 25s the patch passed with JDK v1.7.0_121 +1 javac 1m 25s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 2 new + 242 unchanged - 1 fixed = 244 total (was 243) +1 mvnsite 1m 26s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 1s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_111 +1 javadoc 1m 55s the patch passed with JDK v1.7.0_121 +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121. -1 unit 51m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 169m 13s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.hdfs.server.namenode.TestStartup JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845470/HDFS-11156-branch-2.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a77a6150771d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / db947fb Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18022/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18022/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18022/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18022/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Reopening for precommit run.

          Show
          andrew.wang Andrew Wang added a comment - Reopening for precommit run.
          Hide
          cheersyang Weiwei Yang added a comment -

          Sure Andrew Wang, I just uploaded the patch for branch-2. The failure was because branch-2 uses jdk 7 to build and trunk uses jdk 8 which allows a local class to access local variables and parameters of the enclosing block that are final or effectively final, see more doc here.

          Thank you

          Show
          cheersyang Weiwei Yang added a comment - Sure Andrew Wang , I just uploaded the patch for branch-2. The failure was because branch-2 uses jdk 7 to build and trunk uses jdk 8 which allows a local class to access local variables and parameters of the enclosing block that are final or effectively final , see more doc here . Thank you
          Hide
          andrew.wang Andrew Wang added a comment -

          Sorry about that, I've reverted the branch-2 patch. Weiwei Yang want to provide a branch-2 patch?

          Show
          andrew.wang Andrew Wang added a comment - Sorry about that, I've reverted the branch-2 patch. Weiwei Yang want to provide a branch-2 patch?
          Hide
          daryn Daryn Sharp added a comment -

          This broke branch-2 due to inner class accessing a non-final.

          Show
          daryn Daryn Sharp added a comment - This broke branch-2 due to inner class accessing a non-final.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11063 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11063/)
          HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. (wang: rev 7fcc73fc0d248aae1edbd4e1514c5818f6198928)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11063 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11063/ ) HDFS-11156 . Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. (wang: rev 7fcc73fc0d248aae1edbd4e1514c5818f6198928) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          Hide
          andrew.wang Andrew Wang added a comment -

          I've committed this to trunk and branch-2, thanks for the contribution Weiwei Yang!

          Show
          andrew.wang Andrew Wang added a comment - I've committed this to trunk and branch-2, thanks for the contribution Weiwei Yang !
          Hide
          andrew.wang Andrew Wang added a comment -

          Sorry for the delay, I was on vacation. +1 LGTM, thanks for the hard work and sticking with this JIRA! I'll commit shortly.

          Show
          andrew.wang Andrew Wang added a comment - Sorry for the delay, I was on vacation. +1 LGTM, thanks for the hard work and sticking with this JIRA! I'll commit shortly.
          Hide
          cheersyang Weiwei Yang added a comment -

          Rebase to latest trunk, Andrew Wang please let me know if you have any more comments, thanks.

          Show
          cheersyang Weiwei Yang added a comment - Rebase to latest trunk, Andrew Wang please let me know if you have any more comments, thanks.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Andrew Wang

          When you get a chance, can you help to check if v15 addressed all the comments you gave?

          Thanks a lot.

          Show
          cheersyang Weiwei Yang added a comment - Hello Andrew Wang When you get a chance, can you help to check if v15 addressed all the comments you gave? Thanks a lot.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 20m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 27s Maven dependency ordering for branch
          +1 mvninstall 12m 53s trunk passed
          +1 compile 1m 20s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 22s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 5s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 6s Maven dependency ordering for patch
          +1 mvninstall 1m 14s the patch passed
          +1 compile 1m 17s the patch passed
          +1 javac 1m 17s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 240 unchanged - 1 fixed = 242 total (was 241)
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 0m 55s the patch passed
          +1 unit 0m 53s hadoop-hdfs-client in the patch passed.
          +1 unit 62m 9s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          115m 14s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843534/HDFS-11156.15.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a99d4d671689 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cee0c46
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17876/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17876/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17876/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 20m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 27s Maven dependency ordering for branch +1 mvninstall 12m 53s trunk passed +1 compile 1m 20s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 22s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 6s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 1m 17s the patch passed +1 javac 1m 17s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 240 unchanged - 1 fixed = 242 total (was 241) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 0m 53s hadoop-hdfs-client in the patch passed. +1 unit 62m 9s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 115m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843534/HDFS-11156.15.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a99d4d671689 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cee0c46 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17876/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17876/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17876/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Thanks Andrew Wang for the review. Just uploaded v15 patch including following changes

          1. Completed 4 cases in TestWebHDFS to cover all combinations of offset and length parameters
          2. Added 1 test case to verify that rest call returns an empty array of BlockLoaction when offset is bigger than the file length
          Show
          cheersyang Weiwei Yang added a comment - - edited Thanks Andrew Wang for the review. Just uploaded v15 patch including following changes Completed 4 cases in TestWebHDFS to cover all combinations of offset and length parameters Added 1 test case to verify that rest call returns an empty array of BlockLoaction when offset is bigger than the file length
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the explanation, sounds good. A little nit though:

          +              + "&length=1024&&offset=0");
          

          There are two "&&" here, so I don't think it's testing what we expect. I'd also like to see tests for all four possible combinations of flags.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the explanation, sounds good. A little nit though: + + "&length=1024&&offset=0"); There are two "&&" here, so I don't think it's testing what we expect. I'd also like to see tests for all four possible combinations of flags.
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Hello Andrew Wang

          Please help to review v14 patch when you get a chance, it addressed the comments from you with the changes I described here. The UT failure was not related, and the 2 checkstyle issues were not new as well (I explained ealier).

          Thank you

          Show
          cheersyang Weiwei Yang added a comment - - edited Hello Andrew Wang Please help to review v14 patch when you get a chance, it addressed the comments from you with the changes I described here . The UT failure was not related, and the 2 checkstyle issues were not new as well (I explained ealier). Thank you
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 28s Maven dependency ordering for branch
          +1 mvninstall 8m 50s trunk passed
          +1 compile 1m 24s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 4s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 18s the patch passed
          +1 javac 1m 18s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 240 unchanged - 1 fixed = 242 total (was 241)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 13s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          -1 unit 61m 37s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          90m 31s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestTrashWithSecureEncryptionZones
            hadoop.hdfs.TestSecureEncryptionZoneWithKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842743/HDFS-11156.14.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f0ef09a9fc20 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c38f11
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17828/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17828/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 28s Maven dependency ordering for branch +1 mvninstall 8m 50s trunk passed +1 compile 1m 24s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 18s the patch passed +1 javac 1m 18s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 240 unchanged - 1 fixed = 242 total (was 241) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 13s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. -1 unit 61m 37s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 90m 31s Reason Tests Failed junit tests hadoop.hdfs.TestTrashWithSecureEncryptionZones   hadoop.hdfs.TestSecureEncryptionZoneWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842743/HDFS-11156.14.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f0ef09a9fc20 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c38f11 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17828/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17828/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17828/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17828/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 8m 18s trunk passed
          +1 compile 1m 31s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 43s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 37s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 22s the patch passed
          +1 javac 1m 22s the patch passed
          -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 4 new + 241 unchanged - 1 fixed = 245 total (was 242)
          +1 mvnsite 1m 38s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 34s the patch passed
          +1 javadoc 1m 1s the patch passed
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          -1 unit 86m 9s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          116m 47s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestSecureEncryptionZoneWithKMS
            hadoop.hdfs.TestTrashWithSecureEncryptionZones
            hadoop.hdfs.TestClientProtocolForPipelineRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842697/HDFS-11156.13.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f6457a8db02e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c38f11
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17823/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17823/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17823/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17823/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 8m 18s trunk passed +1 compile 1m 31s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 43s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 37s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 22s the patch passed +1 javac 1m 22s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 4 new + 241 unchanged - 1 fixed = 245 total (was 242) +1 mvnsite 1m 38s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 34s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 0m 56s hadoop-hdfs-client in the patch passed. -1 unit 86m 9s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 116m 47s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestSecureEncryptionZoneWithKMS   hadoop.hdfs.TestTrashWithSecureEncryptionZones   hadoop.hdfs.TestClientProtocolForPipelineRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842697/HDFS-11156.13.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f6457a8db02e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c38f11 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17823/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17823/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17823/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17823/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Andrew Wang

          Thanks for the comments, here

          We don't need to close the FS in NamenodeWebHdfsMethods

          Fixed that in v13 patch.

          offset and length are both required parameters for the FileSystem API. We should require them here too. I don't know why GET_BLOCK_LOCATIONS allows a null length, since they're both required in the PB RPC format too. A test for this would be good.

          Current behavior allows user to submit a http request with or without offset and length parameters, so "op=GET_BLOCK_LOCATIONS" and "op=GET_BLOCK_LOCATIONS&offset=0&length=..." both works. The default value for OffsetParam is 0 but default value for LengthParam is null, I think that was why length has an additional check on null value. I did not change this in v13 patch, otherwise we are not 100% backwards compatible. I added some tests in TestWebHDFS to make sure both forms of rest call works. Hope that is enough for this.

          typo "Loactions" in comment in TestWebHDFS

          Fixed that in v13 patch.

          Thanks.

          Show
          cheersyang Weiwei Yang added a comment - Hello Andrew Wang Thanks for the comments, here We don't need to close the FS in NamenodeWebHdfsMethods Fixed that in v13 patch. offset and length are both required parameters for the FileSystem API. We should require them here too. I don't know why GET_BLOCK_LOCATIONS allows a null length, since they're both required in the PB RPC format too. A test for this would be good. Current behavior allows user to submit a http request with or without offset and length parameters, so "op=GET_BLOCK_LOCATIONS" and "op=GET_BLOCK_LOCATIONS&offset=0&length=..." both works. The default value for OffsetParam is 0 but default value for LengthParam is null, I think that was why length has an additional check on null value. I did not change this in v13 patch, otherwise we are not 100% backwards compatible. I added some tests in TestWebHDFS to make sure both forms of rest call works. Hope that is enough for this. typo "Loactions" in comment in TestWebHDFS Fixed that in v13 patch. Thanks.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the rev Weiwei Yang, looks good overall, some small review comments:

          • We don't need to close the FS in NamenodeWebHdfsMethods. Closing means it gets removed from the FS cache, and then it needs to be re-instantiated next time.
          • offset and length are both required parameters for the FileSystem API. We should require them here too. I don't know why GET_BLOCK_LOCATIONS allows a null length, since they're both required in the PB RPC format too. A test for this would be good.
          • typo "Loactions" in comment in TestWebHDFS
          Show
          andrew.wang Andrew Wang added a comment - Thanks for the rev Weiwei Yang , looks good overall, some small review comments: We don't need to close the FS in NamenodeWebHdfsMethods. Closing means it gets removed from the FS cache, and then it needs to be re-instantiated next time. offset and length are both required parameters for the FileSystem API. We should require them here too. I don't know why GET_BLOCK_LOCATIONS allows a null length, since they're both required in the PB RPC format too. A test for this would be good. typo "Loactions" in comment in TestWebHDFS
          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Andrew Wang

          Please let me know if you have any more comments (to v12 patch), thanks.

          Show
          cheersyang Weiwei Yang added a comment - Hi Andrew Wang Please let me know if you have any more comments (to v12 patch), thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 7m 11s trunk passed
          +1 compile 1m 21s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 8s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 17s the patch passed
          +1 javac 1m 17s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 2 new + 241 unchanged - 1 fixed = 243 total (was 242)
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 18s the patch passed
          +1 javadoc 0m 54s the patch passed
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed.
          +1 unit 85m 28s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          112m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842483/HDFS-11156.12.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7fc0e7d0de82 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 13d8e55
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17804/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17804/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17804/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 7m 11s trunk passed +1 compile 1m 21s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 8s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 17s the patch passed +1 javac 1m 17s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 2 new + 241 unchanged - 1 fixed = 243 total (was 242) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 18s the patch passed +1 javadoc 0m 54s the patch passed +1 unit 0m 55s hadoop-hdfs-client in the patch passed. +1 unit 85m 28s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 112m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842483/HDFS-11156.12.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7fc0e7d0de82 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13d8e55 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17804/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17804/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17804/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 7m 13s trunk passed
          +1 compile 1m 30s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 24s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 1m 23s the patch passed
          +1 javac 1m 23s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 6 new + 241 unchanged - 1 fixed = 247 total (was 242)
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 51s the patch passed
          +1 javadoc 1m 3s the patch passed
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
          -1 unit 94m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          123m 24s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestReadStripedFileWithMissingBlocks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842297/HDFS-11156.11.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b5a9af0e2c82 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9ef89ed
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17796/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17796/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17796/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17796/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 7m 13s trunk passed +1 compile 1m 30s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 24s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 1m 23s the patch passed +1 javac 1m 23s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 6 new + 241 unchanged - 1 fixed = 247 total (was 242) +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 51s the patch passed +1 javadoc 1m 3s the patch passed +1 unit 1m 3s hadoop-hdfs-client in the patch passed. -1 unit 94m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 123m 24s Reason Tests Failed junit tests hadoop.hdfs.TestReadStripedFileWithMissingBlocks Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842297/HDFS-11156.11.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b5a9af0e2c82 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9ef89ed Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17796/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17796/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17796/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17796/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          Attach the patch again to trigger the jenkins job. Last time it did not run because of error No JDK named ?jdk-1.8.0? found.

          Show
          cheersyang Weiwei Yang added a comment - Attach the patch again to trigger the jenkins job. Last time it did not run because of error No JDK named ?jdk-1.8.0? found .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 6s Maven dependency ordering for branch
          +1 mvninstall 8m 29s trunk passed
          +1 compile 1m 42s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 42s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 22s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 1m 27s the patch passed
          +1 javac 1m 27s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 6 new + 241 unchanged - 1 fixed = 247 total (was 242)
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 24s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          +1 unit 93m 17s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          123m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842272/HDFS-11156.10.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1d111476babe 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f54afdb
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17793/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17793/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17793/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 8m 29s trunk passed +1 compile 1m 42s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 42s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 22s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 1m 27s the patch passed +1 javac 1m 27s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 6 new + 241 unchanged - 1 fixed = 247 total (was 242) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 unit 93m 17s hadoop-hdfs in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 123m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842272/HDFS-11156.10.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d111476babe 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f54afdb Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17793/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17793/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17793/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          Ah nice catch Andrew Wang. I got your point, let me make the change and submit another patch. How everything else looks ?

          Show
          cheersyang Weiwei Yang added a comment - Ah nice catch Andrew Wang . I got your point, let me make the change and submit another patch. How everything else looks ?
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Weiwei Yang, if you look at the example for FileStatuses, you'll see that there's this structure:

          {
            "FileStatuses":
            {
              "FileStatus":
              [
                {...},
              ]
            }
          }
          

          Basically, "FileStatuses" is a dictionary, which has a key "FileStatus" which is an array of FileStatus dictionaries.

          In the current output, "BlockLocations" is an array of FileStatus. We need an extra level of wrapping, with a "BlockLocation" key pointing to the array of BlockLocation dictionaries.

          Show
          andrew.wang Andrew Wang added a comment - Hi Weiwei Yang , if you look at the example for FileStatuses , you'll see that there's this structure: { "FileStatuses": { "FileStatus": [ {...}, ] } } Basically, "FileStatuses" is a dictionary, which has a key "FileStatus" which is an array of FileStatus dictionaries. In the current output, "BlockLocations" is an array of FileStatus. We need an extra level of wrapping, with a "BlockLocation" key pointing to the array of BlockLocation dictionaries.
          Hide
          cheersyang Weiwei Yang added a comment -

          The checkstyle issue in v8 patch is not a concern, like I commented before,

          The checkstyle issues were not caused by this patch either, 1st issue was the method too long, 2nd the code style was using nested blocks in switch-case block.

          Show
          cheersyang Weiwei Yang added a comment - The checkstyle issue in v8 patch is not a concern, like I commented before, The checkstyle issues were not caused by this patch either, 1st issue was the method too long, 2nd the code style was using nested blocks in switch-case block.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 7m 27s trunk passed
          +1 compile 1m 28s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 10s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 1m 26s the patch passed
          +1 javac 1m 26s the patch passed
          -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 2 new + 237 unchanged - 0 fixed = 239 total (was 237)
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 29s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          +1 unit 95m 5s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          123m 37s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842172/HDFS-11156.09.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ac0ed24f5c3b 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c73e08a
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17788/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17788/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17788/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 27s trunk passed +1 compile 1m 28s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 1m 26s the patch passed +1 javac 1m 26s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project: The patch generated 2 new + 237 unchanged - 0 fixed = 239 total (was 237) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 29s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 0m 56s hadoop-hdfs-client in the patch passed. +1 unit 95m 5s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 123m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842172/HDFS-11156.09.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ac0ed24f5c3b 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c73e08a Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17788/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17788/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17788/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 30s Maven dependency ordering for branch
          +1 mvninstall 7m 15s trunk passed
          +1 compile 1m 20s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 5s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 18s the patch passed
          +1 javac 1m 18s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 5 new + 237 unchanged - 0 fixed = 242 total (was 237)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          -1 whitespace 0m 0s The patch 19 line(s) with tabs.
          +1 findbugs 3m 18s the patch passed
          -1 javadoc 0m 17s hadoop-hdfs-project_hadoop-hdfs-client generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          +1 unit 61m 9s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          88m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842153/HDFS-11156.08.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6382d8b21382 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 563480d
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17786/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17786/artifact/patchprocess/whitespace-tabs.txt
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17786/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17786/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17786/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 7m 15s trunk passed +1 compile 1m 20s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 18s the patch passed +1 javac 1m 18s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 5 new + 237 unchanged - 0 fixed = 242 total (was 237) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 20s the patch passed -1 whitespace 0m 0s The patch 19 line(s) with tabs. +1 findbugs 3m 18s the patch passed -1 javadoc 0m 17s hadoop-hdfs-project_hadoop-hdfs-client generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1) +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 unit 61m 9s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 88m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842153/HDFS-11156.08.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6382d8b21382 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 563480d Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17786/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17786/artifact/patchprocess/whitespace-tabs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/17786/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17786/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17786/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Hello Andrew Wang

          Thanks for your comments. I just uploaded v8 patch with following changes

          unit test for the fallback behavior

          Added an unit test for the fallback code. Mocked an old version of server, tested 3 scenarios, 1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send back an error. 2) Server handles GET_BLOCK_LOCATIONS and returns a sample of data. 3) Server sends out a random exception.

          separate functions

          I am still using single function to handle that, but I added more java doc ahead. The reason is 1) Separate them would have majority of code similar 2) I checked the times of this function gets called while doing the fallback UT, this is easier with single method.

          The example curl and response are indented by 9 instead of 8

          I am following the indents format with rest of the document. They are also 9. Anyhow I checked the indents again in the doc I added, it should be OK now.

          provide an example response for a file with more than one block

          Done

          I don't think this schema works

          I verified the schema again with this online tool, and compared with FileStatuses/FileStatus schemas (they are similar). See Sample response, BlockLocations JSON schema which looks just like FileStatuses JSON schema, and BlockLocation Properties Schema . Let me if it looks good now.

          Thank you and let me know if you have any further comments.

          Show
          cheersyang Weiwei Yang added a comment - - edited Hello Andrew Wang Thanks for your comments. I just uploaded v8 patch with following changes unit test for the fallback behavior Added an unit test for the fallback code. Mocked an old version of server, tested 3 scenarios, 1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send back an error. 2) Server handles GET_BLOCK_LOCATIONS and returns a sample of data. 3) Server sends out a random exception. separate functions I am still using single function to handle that, but I added more java doc ahead. The reason is 1) Separate them would have majority of code similar 2) I checked the times of this function gets called while doing the fallback UT, this is easier with single method. The example curl and response are indented by 9 instead of 8 I am following the indents format with rest of the document. They are also 9. Anyhow I checked the indents again in the doc I added, it should be OK now. provide an example response for a file with more than one block Done I don't think this schema works I verified the schema again with this online tool , and compared with FileStatuses/FileStatus schemas (they are similar). See Sample response , BlockLocations JSON schema which looks just like FileStatuses JSON schema , and BlockLocation Properties Schema . Let me if it looks good now. Thank you and let me know if you have any further comments.
          Hide
          andrew.wang Andrew Wang added a comment -

          Looks pretty good, thanks for revving. A few comments:

          • Is it possible to do a unit test for the fallback behavior, by mocking the server to throw the exception? Can also try throwing a different exception to make sure it doesn't erroneously fallback.
          • Can you make separate functions for doing GETFILEBLOCKLOCATIONS and GET_BLOCK_LOCATIONS? I'd prefer that since it's simpler for the caller and then we don't need error checking for an unknown operation.
          • The example curl and response are indented by 9 instead of 8
          • In the doc, can you provide an example response for a file with more than one block?
          • I don't think this schema works, since there should be a nested "BlockLocation" inside the "BlockLocations" and an array somewhere to handle multiple blocks. See FileStatuses and FileStatus as an example.
          Show
          andrew.wang Andrew Wang added a comment - Looks pretty good, thanks for revving. A few comments: Is it possible to do a unit test for the fallback behavior, by mocking the server to throw the exception? Can also try throwing a different exception to make sure it doesn't erroneously fallback. Can you make separate functions for doing GETFILEBLOCKLOCATIONS and GET_BLOCK_LOCATIONS? I'd prefer that since it's simpler for the caller and then we don't need error checking for an unknown operation. The example curl and response are indented by 9 instead of 8 In the doc, can you provide an example response for a file with more than one block? I don't think this schema works, since there should be a nested "BlockLocation" inside the "BlockLocations" and an array somewhere to handle multiple blocks. See FileStatuses and FileStatus as an example.
          Hide
          cheersyang Weiwei Yang added a comment -

          Uploaded v7 patch, it includes

          1. V6 patch in this ticket
          2. The doc patch in HDFS-11166
          3. Added a fall back mechanism to handle the backwards compatibility.

          I set up a 2.7.2 cluster, and tested to use trunk client library to access webhdfs on the 2.7.2 cluster, without the #3 fix, it failed with following exception

          [yangww@host1 hadoop-3.0.0-alpha2-SNAPSHOT]$ ./bin/hadoop jar /tmp/WebhdfsClient.jar WebhdfsClient
          16/12/06 06:43:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
          org.apache.hadoop.ipc.RemoteException(java.lang.IllegalArgumentException): Invalid value for webhdfs parameter "op": No enum constant org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETFILEBLOCKLOCATIONS
          	at org.apache.hadoop.hdfs.web.JsonUtilClient.toRemoteException(JsonUtilClient.java:73)
          	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:474)
          

          with the patch, it falls back to use GET_BLOCK_LOCATIONS, output looks like

          [yangww@host1 hadoop-3.0.0-alpha2-SNAPSHOT]$ ./bin/hadoop jar /tmp/WebhdfsClient.jar WebhdfsClient
          16/12/06 07:36:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
          16/12/06 07:36:25 INFO web.WebHdfsFileSystem: Invalid webhdfs operation parameter GETFILEBLOCKLOCATIONS.Fallback to use GET_BLOCK_LOCATIONS
          ... block locations ...
          
          Show
          cheersyang Weiwei Yang added a comment - Uploaded v7 patch, it includes V6 patch in this ticket The doc patch in HDFS-11166 Added a fall back mechanism to handle the backwards compatibility. I set up a 2.7.2 cluster, and tested to use trunk client library to access webhdfs on the 2.7.2 cluster, without the #3 fix, it failed with following exception [yangww@host1 hadoop-3.0.0-alpha2-SNAPSHOT]$ ./bin/hadoop jar /tmp/WebhdfsClient.jar WebhdfsClient 16/12/06 06:43:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable org.apache.hadoop.ipc.RemoteException(java.lang.IllegalArgumentException): Invalid value for webhdfs parameter "op": No enum constant org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETFILEBLOCKLOCATIONS at org.apache.hadoop.hdfs.web.JsonUtilClient.toRemoteException(JsonUtilClient.java:73) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:474) with the patch, it falls back to use GET_BLOCK_LOCATIONS, output looks like [yangww@host1 hadoop-3.0.0-alpha2-SNAPSHOT]$ ./bin/hadoop jar /tmp/WebhdfsClient.jar WebhdfsClient 16/12/06 07:36:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/12/06 07:36:25 INFO web.WebHdfsFileSystem: Invalid webhdfs operation parameter GETFILEBLOCKLOCATIONS.Fallback to use GET_BLOCK_LOCATIONS ... block locations ...
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10946 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10946/)
          Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST (wang: rev 08a7253bc0eb6c9155457feecb9c5cdc17c3a814)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10946 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10946/ ) Revert " HDFS-11156 . Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST (wang: rev 08a7253bc0eb6c9155457feecb9c5cdc17c3a814) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          Hide
          cheersyang Weiwei Yang added a comment -

          Sure, thanks a lot.

          Show
          cheersyang Weiwei Yang added a comment - Sure, thanks a lot.
          Hide
          liuml07 Mingliang Liu added a comment -

          Sorry I missed the point of not changing the WebHdfsFileSystem. I think Andrew is making a good point. Please take care of the revert. Thanks,

          Show
          liuml07 Mingliang Liu added a comment - Sorry I missed the point of not changing the WebHdfsFileSystem. I think Andrew is making a good point. Please take care of the revert. Thanks,
          Hide
          andrew.wang Andrew Wang added a comment -

          Great, thanks Weiwei Yang. I'm going to revert this in the meantime.

          Also, could you fold the doc improvements in HDFS-11166 into this patch? Normally we commit the code and docs at the same time.

          Show
          andrew.wang Andrew Wang added a comment - Great, thanks Weiwei Yang . I'm going to revert this in the meantime. Also, could you fold the doc improvements in HDFS-11166 into this patch? Normally we commit the code and docs at the same time.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Andrew Wang

          You are right. I missed that. Thanks!
          A fallback would be nice, looks like the fallback should only happen when the client gets a "UnsupportedOperationException - GETFILEBLOCKLOCATIONS is not supported" from old version of webhdfs server. Let me work on this.

          Show
          cheersyang Weiwei Yang added a comment - Hi Andrew Wang You are right. I missed that. Thanks! A fallback would be nice, looks like the fallback should only happen when the client gets a "UnsupportedOperationException - GETFILEBLOCKLOCATIONS is not supported" from old version of webhdfs server. Let me work on this.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Weiwei Yang I think you misunderstand: the new version of WebHdfsFileSystem#getFileBlockLocations always calls GETFILEBLOCKLOCATIONS, there is no fallback mechanism to call GET_BLOCK_LOCATIONS. I'm basically proposing we add a fallback to the old code.

          Looking at HDFS-10756, it implements a fallback to the default implementation if the call fails:

          1638	    } catch(IOException e) {
          1639	      LOG.warn("Cannot find trash root of " + path, e);
          1640	      // keep the same behavior with dfs
          1641	      return super.getTrashRoot(path).makeQualified(getUri(), null);
          1642	    }
          
          Show
          andrew.wang Andrew Wang added a comment - Hi Weiwei Yang I think you misunderstand: the new version of WebHdfsFileSystem#getFileBlockLocations always calls GETFILEBLOCKLOCATIONS, there is no fallback mechanism to call GET_BLOCK_LOCATIONS. I'm basically proposing we add a fallback to the old code. Looking at HDFS-10756 , it implements a fallback to the default implementation if the call fails: 1638 } catch (IOException e) { 1639 LOG.warn( "Cannot find trash root of " + path, e); 1640 // keep the same behavior with dfs 1641 return super .getTrashRoot(path).makeQualified(getUri(), null ); 1642 }
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Hi Andrew Wang

          In that case client can still call GET_BLOCK_LOCATIONS, that is still working. Adding a new API should not break compatibility, this one is jus like HDFS-10756 which adds "GETTRASHROOT" to webhdfs. What do you think?

          Show
          cheersyang Weiwei Yang added a comment - - edited Hi Andrew Wang In that case client can still call GET_BLOCK_LOCATIONS, that is still working. Adding a new API should not break compatibility, this one is jus like HDFS-10756 which adds "GETTRASHROOT" to webhdfs. What do you think?
          Hide
          andrew.wang Andrew Wang added a comment -

          To expand on my previous comment, the new client will try to call GETFILEBLOCKLOCATIONS in WebHdfsFileSystem, which will not be implemented on an old cluster.

          Show
          andrew.wang Andrew Wang added a comment - To expand on my previous comment, the new client will try to call GETFILEBLOCKLOCATIONS in WebHdfsFileSystem, which will not be implemented on an old cluster.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Andrew Wang

          Thank you for raising up your concern. I may not fully get your point, I think we are maintaining the compatibility here. Webhdfs GET_BLOCK_LOCATIONS is still working as before, see this comment , and WebHdfsFileSystem#getFileBlockLocations is still returning BlockLocation[] as before, nothing really changes. Can you please give me some more detail what would be broken? I can test the scenario in advance.

          Thanks

          Show
          cheersyang Weiwei Yang added a comment - Hello Andrew Wang Thank you for raising up your concern. I may not fully get your point, I think we are maintaining the compatibility here. Webhdfs GET_BLOCK_LOCATIONS is still working as before, see this comment , and WebHdfsFileSystem#getFileBlockLocations is still returning BlockLocation[] as before, nothing really changes. Can you please give me some more detail what would be broken? I can test the scenario in advance. Thanks
          Hide
          andrew.wang Andrew Wang added a comment -

          Sorry I didn't get a chance to review this earlier, but I have a possible compatibility concern. As I described earlier, webhdfs compatibility is important so we can move data from an older to a newer cluster with distcp. distcp is also often run in "pull" mode, with a new client on the new cluster reading from the old cluster. See these tables which recommend running on the destination cluster:

          https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_Sys_Admin_Guides/content/ref-cfb69f75-d06f-46a2-862f-efeba959b152.1.html
          https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_admin_distcp_data_cluster_migrate.html

          Since we don't have a fallback to use GET_BLOCK_LOCATIONS instead, getFileBlockLocations won't work with the new client/old cluster case.

          Weiwei Yang, Mingliang Liu what do you think? Wondering if we should revert to handle this fallback.

          Show
          andrew.wang Andrew Wang added a comment - Sorry I didn't get a chance to review this earlier, but I have a possible compatibility concern. As I described earlier, webhdfs compatibility is important so we can move data from an older to a newer cluster with distcp. distcp is also often run in "pull" mode, with a new client on the new cluster reading from the old cluster. See these tables which recommend running on the destination cluster: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_Sys_Admin_Guides/content/ref-cfb69f75-d06f-46a2-862f-efeba959b152.1.html https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_admin_distcp_data_cluster_migrate.html Since we don't have a fallback to use GET_BLOCK_LOCATIONS instead, getFileBlockLocations won't work with the new client/old cluster case. Weiwei Yang , Mingliang Liu what do you think? Wondering if we should revert to handle this fallback.
          Hide
          cheersyang Weiwei Yang added a comment -
          Show
          cheersyang Weiwei Yang added a comment - Thanks Mingliang Liu
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Committed to trunk through branch-2.8 branches; fixed trivial cherry-pick conflicts.

          Show
          liuml07 Mingliang Liu added a comment - +1 Committed to trunk through branch-2.8 branches; fixed trivial cherry-pick conflicts.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10930 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10930/)
          HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. (liuml07: rev c7ff34f8dcca3a2024230c5383abd9299daa1b20)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10930 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10930/ ) HDFS-11156 . Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. (liuml07: rev c7ff34f8dcca3a2024230c5383abd9299daa1b20) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 30s Maven dependency ordering for branch
          +1 mvninstall 8m 27s trunk passed
          +1 compile 1m 38s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 44s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          -1 findbugs 2m 0s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 1m 7s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 33s the patch passed
          +1 compile 1m 33s the patch passed
          +1 javac 1m 33s the patch passed
          -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 2 new + 237 unchanged - 0 fixed = 239 total (was 237)
          +1 mvnsite 1m 30s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 4s the patch passed
          +1 javadoc 1m 2s the patch passed
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
          -1 unit 70m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          102m 20s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841521/HDFS-11156.06.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c4f8def85373 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c87b3a4
          Default Java 1.8.0_111
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17744/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17744/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17744/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17744/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17744/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 8m 27s trunk passed +1 compile 1m 38s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 44s trunk passed +1 mvneclipse 0m 27s trunk passed -1 findbugs 2m 0s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 7s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed +1 compile 1m 33s the patch passed +1 javac 1m 33s the patch passed -0 checkstyle 0m 32s hadoop-hdfs-project: The patch generated 2 new + 237 unchanged - 0 fixed = 239 total (was 237) +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 4s the patch passed +1 javadoc 1m 2s the patch passed +1 unit 1m 3s hadoop-hdfs-client in the patch passed. -1 unit 70m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 102m 20s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841521/HDFS-11156.06.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4f8def85373 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c87b3a4 Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17744/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17744/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17744/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17744/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17744/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          Uploaded v6 patch to address a minor change on the comments for "GET_BLOCK_LOCATIONS" according to the discussion here

          Show
          cheersyang Weiwei Yang added a comment - Uploaded v6 patch to address a minor change on the comments for "GET_BLOCK_LOCATIONS" according to the discussion here
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Mingliang Liu

          Thanks, actually I created HDFS-11166 along with this one to track the docs update, since this one has no change to GET_BLOCK_LOCATIONS, that doc JIRA needs also be updated. I will add the required docs from that JIRA, is that OK ?

          Show
          cheersyang Weiwei Yang added a comment - Hello Mingliang Liu Thanks, actually I created HDFS-11166 along with this one to track the docs update, since this one has no change to GET_BLOCK_LOCATIONS, that doc JIRA needs also be updated. I will add the required docs from that JIRA, is that OK ?
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Please also add doc in the REST API page hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md; also add release note for this JIRA. I changed its type from BUG to Improvement.

          After that, I will commit if no more comment by tomorrow.

          Thanks all for the helpful discussion.

          Show
          liuml07 Mingliang Liu added a comment - +1 Please also add doc in the REST API page hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md ; also add release note for this JIRA. I changed its type from BUG to Improvement . After that, I will commit if no more comment by tomorrow. Thanks all for the helpful discussion.
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Regarding to the jenkins job result

          1. The findbugs error was from a recent commit HDFS-10930, not from this patch.
          2. The checkstyle issues were not caused by this patch either, 1st issue was the method too long, 2nd the code style was using nested blocks in switch-case block.
          3. The 2 UT failures also not related, I have run them locally and they all succeed. They seemed to be intermittent failures, HDFS-9599 and HDFS-11030 might be related.

          I've also tested this with latest trunk code on a small cluster, both

          http://<nn>:<port>/webhdfs/v1/tmp/test.log?op=GETFILEBLOCKLOCATIONS&length=100&offset=0
          

          and

          http://<nn>:<port>/webhdfs/v1/tmp/test.log?op=GET_BLOCK_LOCATIONS&length=100&offset=0
          

          works and returned expected value. Mingliang Liu Would you please help to review the patch?

          Thanks a lot.

          Show
          cheersyang Weiwei Yang added a comment - - edited Regarding to the jenkins job result The findbugs error was from a recent commit HDFS-10930 , not from this patch. The checkstyle issues were not caused by this patch either, 1st issue was the method too long, 2nd the code style was using nested blocks in switch-case block. The 2 UT failures also not related, I have run them locally and they all succeed. They seemed to be intermittent failures, HDFS-9599 and HDFS-11030 might be related. I've also tested this with latest trunk code on a small cluster, both http://<nn>:<port>/webhdfs/v1/tmp/test.log?op=GETFILEBLOCKLOCATIONS&length=100&offset=0 and http://<nn>:<port>/webhdfs/v1/tmp/test.log?op=GET_BLOCK_LOCATIONS&length=100&offset=0 works and returned expected value. Mingliang Liu Would you please help to review the patch? Thanks a lot.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 7m 28s trunk passed
          +1 compile 1m 26s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 32s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 1m 21s the patch passed
          +1 javac 1m 21s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 2 new + 237 unchanged - 0 fixed = 239 total (was 237)
          +1 mvnsite 1m 31s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 39s the patch passed
          +1 javadoc 0m 58s the patch passed
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed.
          -1 unit 102m 9s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          130m 47s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestDecommissioningStatus



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841220/HDFS-11156.05.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0d7e5a7f263a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1f7613b
          Default Java 1.8.0_111
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17725/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17725/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17725/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17725/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 7m 28s trunk passed +1 compile 1m 26s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 25s trunk passed -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 1m 21s the patch passed +1 javac 1m 21s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project: The patch generated 2 new + 237 unchanged - 0 fixed = 239 total (was 237) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 39s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 1m 1s hadoop-hdfs-client in the patch passed. -1 unit 102m 9s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 130m 47s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestDecommissioningStatus Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841220/HDFS-11156.05.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0d7e5a7f263a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1f7613b Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17725/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17725/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17725/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17725/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17725/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          this API returns BlockLocations[] + FileStatus, it has more information but also means more work for clients to parse and more stuff on network. getFileBlockLocations API should be good in most cases.

          I'm guessing your client already has code to parse a FileStatus, so is this really more work? This depends on your usecase, but often the client will want a FileStatus too.

          Anyway, I'm okay with a GETFILEBLOCKLOCATIONS API if that's what fits best. Thanks for working on this Weiwei Yang!

          Show
          andrew.wang Andrew Wang added a comment - this API returns BlockLocations[] + FileStatus, it has more information but also means more work for clients to parse and more stuff on network. getFileBlockLocations API should be good in most cases. I'm guessing your client already has code to parse a FileStatus, so is this really more work? This depends on your usecase, but often the client will want a FileStatus too. Anyway, I'm okay with a GETFILEBLOCKLOCATIONS API if that's what fits best. Thanks for working on this Weiwei Yang !
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Tsz Wo Nicholas Sze

          Thanks for the input, I am happy with that .
          Thanks for Mingliang Liu and Andrew Wang to join the discussion, appreciate your help .
          I'll upload a patch tomorrow since most of work has already done in previous patch. I have also removed "incompatible change" tag.

          Show
          cheersyang Weiwei Yang added a comment - Hello Tsz Wo Nicholas Sze Thanks for the input, I am happy with that . Thanks for Mingliang Liu and Andrew Wang to join the discussion, appreciate your help . I'll upload a patch tomorrow since most of work has already done in previous patch. I have also removed "incompatible change" tag.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The use of underscore in GET_BLOCK_LOCATIONS is to emphasize that it is an internal API.

          I agree with Mingliang Liu's suggestion that to

          1. Provide a new OP GETFILEBLOCKLOCATIONS for webhdfs that returns BlockLocation[] to comply with FileSystem API

          +1 on adding a new GETFILEBLOCKLOCATIONS for FileSystem#getFileBlockLocations and keep GET_BLOCK_LOCATIONS unchanged. Thanks for working on this, Weiwei Yang.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The use of underscore in GET_BLOCK_LOCATIONS is to emphasize that it is an internal API. I agree with Mingliang Liu's suggestion that to Provide a new OP GETFILEBLOCKLOCATIONS for webhdfs that returns BlockLocation[] to comply with FileSystem API +1 on adding a new GETFILEBLOCKLOCATIONS for FileSystem#getFileBlockLocations and keep GET_BLOCK_LOCATIONS unchanged. Thanks for working on this, Weiwei Yang .
          Hide
          cheersyang Weiwei Yang added a comment -

          Hello Mingliang Liu, Andrew Wang

          I agree with Mingliang Liu's suggestion that to

          1. Provide a new OP GETFILEBLOCKLOCATIONS for webhdfs that returns BlockLocation[] to comply with FileSystem API
          2. Add documentation for GETFILEBLOCKLOCATIONS for webhdfs, along with the info of its response json type,

          In our case, client is trying to switch their application from calling java API to webhdfs, then failed to find the equivalent API of getFileBlockLocations. No doc provided, we figured by reading the source code but end up with an unexpected output from current GET_BLOCK_LOCATIONS. This is not user-friendly. I am going to remove the tag of "Incompatible changes" because adding a new API with well documented will maintain the compatibility, and provide a FileSystem compliant way to user to get block locations via rest api.

          Further more, what's the meaning of keep a "private unstable op" in web hdfs? Webhdfs APIs are not private, you can't stop user calling them. If it is considered as unstable, how about something in document

          Get File Block Locations
          
          Submit a HTTP GET request 
          curl -i -L "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GET_BLOCK_LOCATIONS
          
          ... 
          
          Deprecated : use GETFILEBLOCKLOCATIONS instead.
          

          To Andrew's suggestion :

          have you also considered implementing listLocatedStatus, which is IMO the better API since it returns both listing and locations in a single call?

          this API returns BlockLocations[] + FileStatus, it has more information but also means more work for clients to parse and more stuff on network. getFileBlockLocations API should be good in most cases.

          Show
          cheersyang Weiwei Yang added a comment - Hello Mingliang Liu , Andrew Wang I agree with Mingliang Liu 's suggestion that to Provide a new OP GETFILEBLOCKLOCATIONS for webhdfs that returns BlockLocation[] to comply with FileSystem API Add documentation for GETFILEBLOCKLOCATIONS for webhdfs, along with the info of its response json type, In our case, client is trying to switch their application from calling java API to webhdfs, then failed to find the equivalent API of getFileBlockLocations . No doc provided, we figured by reading the source code but end up with an unexpected output from current GET_BLOCK_LOCATIONS. This is not user-friendly. I am going to remove the tag of "Incompatible changes" because adding a new API with well documented will maintain the compatibility, and provide a FileSystem compliant way to user to get block locations via rest api. Further more, what's the meaning of keep a "private unstable op" in web hdfs? Webhdfs APIs are not private, you can't stop user calling them. If it is considered as unstable, how about something in document Get File Block Locations Submit a HTTP GET request curl -i -L "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GET_BLOCK_LOCATIONS ... Deprecated : use GETFILEBLOCKLOCATIONS instead. To Andrew's suggestion : have you also considered implementing listLocatedStatus, which is IMO the better API since it returns both listing and locations in a single call? this API returns BlockLocations[] + FileStatus, it has more information but also means more work for clients to parse and more stuff on network. getFileBlockLocations API should be good in most cases.
          Hide
          andrew.wang Andrew Wang added a comment -

          I looked at the git blame, and HDFS-2527 which added the private/unstable annotation also did a rename from GETFILEBLOCKLOCATIONS to GET_BLOCK_LOCATIONS. I'm guessing it wasn't exposed initially to keep the public WebHDFS API as narrow as possible. Tsz Wo Nicholas Sze do you have any historical perspective on this?

          Could you also provide more information about your usecase? Is there an out-of-tree client you are maintaining? Also, given that GET_BLOCK_LOCATIONS is effectively stable, can you just use that instead (like the Java WebHdfsFileSystem implementation)?

          If we do need to add a new API (which I'm okay with in the abstract), have you also considered implementing listLocatedStatus, which is IMO the better API since it returns both listing and locations in a single call?

          Show
          andrew.wang Andrew Wang added a comment - I looked at the git blame, and HDFS-2527 which added the private/unstable annotation also did a rename from GETFILEBLOCKLOCATIONS to GET_BLOCK_LOCATIONS. I'm guessing it wasn't exposed initially to keep the public WebHDFS API as narrow as possible. Tsz Wo Nicholas Sze do you have any historical perspective on this? Could you also provide more information about your usecase? Is there an out-of-tree client you are maintaining? Also, given that GET_BLOCK_LOCATIONS is effectively stable, can you just use that instead (like the Java WebHdfsFileSystem implementation)? If we do need to add a new API (which I'm okay with in the abstract), have you also considered implementing listLocatedStatus, which is IMO the better API since it returns both listing and locations in a single call?
          Hide
          liuml07 Mingliang Liu added a comment -

          They are two different problems.

          P1. The mismatch of REST getFileBlockLocations with FileSystem API is a public/well-known feature which was designed to be this.

          The general contract of REST API, as Weiwei pointed out above, is to support the FileSystem interface. So I believe this is not the case. Moreover, mismatching APIs should be well-documented which is extremely useful to new users. GET_BLOCK_LOCATIONS is a private unstable op per its documentation, see here. It's even not on the page of WebHDFS REST API documentation.

          P2. The mismatch was an omission but we don't want to fix it for the sake of wire compatibility.

          I can understand that our users may have been using this and it's considered to some degree public/stable. I agree with the community decision that we should break the wire compatibility only for a good reason. I propose that we create a new OP named GETFILEBLOCKLOCATIONS for REST API. The name pattern follows other REST APIs (no underscores); it returns BlockLocations[] as other FileSystem does. WebHdfsFileSystem should not be affected as it returns BlockLocation[] anyway.

          Thanks,

          Show
          liuml07 Mingliang Liu added a comment - They are two different problems. P1. The mismatch of REST getFileBlockLocations with FileSystem API is a public/well-known feature which was designed to be this. The general contract of REST API, as Weiwei pointed out above, is to support the FileSystem interface. So I believe this is not the case. Moreover, mismatching APIs should be well-documented which is extremely useful to new users. GET_BLOCK_LOCATIONS is a private unstable op per its documentation, see here . It's even not on the page of WebHDFS REST API documentation. P2. The mismatch was an omission but we don't want to fix it for the sake of wire compatibility. I can understand that our users may have been using this and it's considered to some degree public/stable. I agree with the community decision that we should break the wire compatibility only for a good reason. I propose that we create a new OP named GETFILEBLOCKLOCATIONS for REST API. The name pattern follows other REST APIs (no underscores); it returns BlockLocations[] as other FileSystem does. WebHdfsFileSystem should not be affected as it returns BlockLocation[] anyway. Thanks,
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 8m 16s trunk passed
          +1 compile 1m 28s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 32s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 40s trunk passed
          +1 javadoc 1m 9s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 1m 32s the patch passed
          +1 javac 1m 32s the patch passed
          +1 checkstyle 0m 34s hadoop-hdfs-project: The patch generated 0 new + 230 unchanged - 4 fixed = 230 total (was 234)
          +1 mvnsite 1m 39s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 52s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 69m 10s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          100m 7s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840821/HDFS-11156.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c9648c977431 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67d9f28
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17694/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17694/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 8m 16s trunk passed +1 compile 1m 28s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 40s trunk passed +1 javadoc 1m 9s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 1m 32s the patch passed +1 javac 1m 32s the patch passed +1 checkstyle 0m 34s hadoop-hdfs-project: The patch generated 0 new + 230 unchanged - 4 fixed = 230 total (was 234) +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 52s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 69m 10s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 100m 7s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840821/HDFS-11156.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c9648c977431 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67d9f28 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17694/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17694/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Hello Andrew Wang

          Thanks for providing your thought.
          From the webhdfs document

          The HTTP REST API supports the complete FileSystem/FileContext interface for HDFS
          

          user would expect the BlockLocation[] instead of LocatedBlocks according to FileSystem#getFileBlockLocations interface. I checked rest of APIs, they seemed good like

          GETFILESTATUS (see FileSystem.getFileStatus)
          

          Isn't this a bug worth to fix? I am hoping you can reconsider this.

          Thank you

          Show
          cheersyang Weiwei Yang added a comment - - edited Hello Andrew Wang Thanks for providing your thought. From the webhdfs document The HTTP REST API supports the complete FileSystem/FileContext interface for HDFS user would expect the BlockLocation[] instead of LocatedBlocks according to FileSystem#getFileBlockLocations interface. I checked rest of APIs, they seemed good like GETFILESTATUS (see FileSystem.getFileStatus) Isn't this a bug worth to fix? I am hoping you can reconsider this. Thank you
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi folks, I'm -1 on this change. Our users rely on WebHDFS in particular for compatibility across major versions. In general, we should not break wire compatibility unless there are very good reasons, and we're trying to preserve wire compatibility even for Hadoop 3.

          WebHDFS also does not expose the FileSystem interface, it exposes the HDFS interface. This means we sometimes return extra stuff like what we have in LocatedBlock vs the generic BlockLocation. This explains the mismatch with the FileSystem API.

          I'd like to close this as WONTFIX if you agree, thanks.

          Show
          andrew.wang Andrew Wang added a comment - Hi folks, I'm -1 on this change. Our users rely on WebHDFS in particular for compatibility across major versions. In general, we should not break wire compatibility unless there are very good reasons, and we're trying to preserve wire compatibility even for Hadoop 3. WebHDFS also does not expose the FileSystem interface, it exposes the HDFS interface. This means we sometimes return extra stuff like what we have in LocatedBlock vs the generic BlockLocation. This explains the mismatch with the FileSystem API. I'd like to close this as WONTFIX if you agree, thanks.
          Hide
          liuml07 Mingliang Liu added a comment -

          The patch looks good to me overall. +1

          I have several minor comments:

          1. Do we need to specially handle the case of list.isEmpty()? Looks pretty similar to the else clause.
            593	    } else if (list.isEmpty()) {
            594	      return new String[0];
            595	    } else {
            596	      final String[] array = new String[list.size()];
            597	      int i = 0;
            598	      for (Object object : list) {
            599	        array[i++] = object.toString();
            600	      }
            601	      return array;
            602	    }
            
          2. Are you considering using assertArrayEquals()?
            testWebHdfsGetBlockLocationsWithStorageType()
            899	        for(int j=0; j<raw.getStorageTypes().length; j++) {
            900	          Assert.assertEquals(
            901	              raw.getStorageTypes()[j],
            902	              rest.getStorageTypes()[j]);
            903	        }
            
          3. This helper method can be static, and the intermediate variable response definition can be joined with the return statement return IOUtils.toString(conn.getInputStream());.
            testWebHdfsGetBlockLocationsWithStorageType()
            912	  private String getResponse(URL url, String httpRequestType)
            913	      throws IOException {
            914	    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            915	    conn.setRequestMethod(httpRequestType);
            916	    conn.setInstanceFollowRedirects(false);
            917	    String response = IOUtils.toString(conn.getInputStream());
            918	    return response;
            919	  }
            
          Show
          liuml07 Mingliang Liu added a comment - The patch looks good to me overall. +1 I have several minor comments: Do we need to specially handle the case of list.isEmpty() ? Looks pretty similar to the else clause. 593 } else if (list.isEmpty()) { 594 return new String [0]; 595 } else { 596 final String [] array = new String [list.size()]; 597 int i = 0; 598 for ( Object object : list) { 599 array[i++] = object.toString(); 600 } 601 return array; 602 } Are you considering using assertArrayEquals() ? testWebHdfsGetBlockLocationsWithStorageType() 899 for ( int j=0; j<raw.getStorageTypes().length; j++) { 900 Assert.assertEquals( 901 raw.getStorageTypes()[j], 902 rest .getStorageTypes()[j]); 903 } This helper method can be static, and the intermediate variable response definition can be joined with the return statement return IOUtils.toString(conn.getInputStream()); . testWebHdfsGetBlockLocationsWithStorageType() 912 private String getResponse(URL url, String httpRequestType) 913 throws IOException { 914 HttpURLConnection conn = (HttpURLConnection) url.openConnection(); 915 conn.setRequestMethod(httpRequestType); 916 conn.setInstanceFollowRedirects( false ); 917 String response = IOUtils.toString(conn.getInputStream()); 918 return response; 919 }
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks a lot Mingliang Liu

          Show
          cheersyang Weiwei Yang added a comment - Thanks a lot Mingliang Liu
          Hide
          liuml07 Mingliang Liu added a comment -

          It may happen this week or after the holiday. Thanks,

          Show
          liuml07 Mingliang Liu added a comment - It may happen this week or after the holiday. Thanks,
          Hide
          cheersyang Weiwei Yang added a comment - - edited

          Hello Mingliang Liu

          Would you help to review this patch ?
          I have built latest trunk and tested with this patch, use

          http://<HOST>:<PORT>/webhdfs/v1/tmp/test1.img?op=GET_BLOCK_LOCATIONS&offset=0&length=1&user.name=yangww
          

          it returns output

          {"BlockLocations":[{"cachedHosts":[],"corrupt":false,"hosts":["deale1.fyre.ibm.com"],"length":134217728,"names":["9.30.167.122:9866"],"offset":0,"storageIds":["DS-2c136bcd-d658-4853-8b7d-9ecb354e6093"],"storageTypes":["DISK"],"topologyPaths":["/default-rack/9.30.167.122:9866"]}]}
          

          client would easily parse this to BlockLocation array using json parser or object mapper, just like the test case code I added to TestWebHDFS.

          Show
          cheersyang Weiwei Yang added a comment - - edited Hello Mingliang Liu Would you help to review this patch ? I have built latest trunk and tested with this patch, use http: //<HOST>:<PORT>/webhdfs/v1/tmp/test1.img?op=GET_BLOCK_LOCATIONS&offset=0&length=1&user.name=yangww it returns output { "BlockLocations" :[{ "cachedHosts" :[], "corrupt" : false , "hosts" :[ "deale1.fyre.ibm.com" ], "length" :134217728, "names" :[ "9.30.167.122:9866" ], "offset" :0, "storageIds" :[ "DS-2c136bcd-d658-4853-8b7d-9ecb354e6093" ], "storageTypes" :[ "DISK" ], "topologyPaths" :[ "/ default -rack/9.30.167.122:9866" ]}]} client would easily parse this to BlockLocation array using json parser or object mapper, just like the test case code I added to TestWebHDFS .
          Hide
          cheersyang Weiwei Yang added a comment -

          I don't think the test failures in latest jenkins job related to this patch, I run them successfully locally, also 1st jenkins job UT was clean, v2/v3 patch were just checkstyle fixes.

          Show
          cheersyang Weiwei Yang added a comment - I don't think the test failures in latest jenkins job related to this patch, I run them successfully locally, also 1st jenkins job UT was clean, v2/v3 patch were just checkstyle fixes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 30s Maven dependency ordering for branch
          +1 mvninstall 7m 59s trunk passed
          +1 compile 1m 25s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 5s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 14s the patch passed
          +1 compile 1m 18s the patch passed
          +1 javac 1m 18s the patch passed
          +1 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 0 new + 231 unchanged - 4 fixed = 231 total (was 235)
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 18s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          -1 unit 65m 37s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          93m 47s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
            hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840030/HDFS-11156.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1fc06549b7a4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a926f89
          Default Java 1.8.0_111
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17633/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17633/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 7m 59s trunk passed +1 compile 1m 25s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 1m 18s the patch passed +1 javac 1m 18s the patch passed +1 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 0 new + 231 unchanged - 4 fixed = 231 total (was 235) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 18s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 0m 54s hadoop-hdfs-client in the patch passed. -1 unit 65m 37s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 93m 47s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840030/HDFS-11156.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1fc06549b7a4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a926f89 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17633/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17633/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 7s Maven dependency ordering for branch
          +1 mvninstall 8m 35s trunk passed
          +1 compile 1m 27s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 33s trunk passed
          +1 javadoc 1m 7s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 34s the patch passed
          +1 compile 1m 39s the patch passed
          +1 javac 1m 39s the patch passed
          -0 checkstyle 0m 34s hadoop-hdfs-project: The patch generated 1 new + 230 unchanged - 4 fixed = 231 total (was 234)
          +1 mvnsite 1m 41s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 23s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 0m 56s hadoop-hdfs-client in the patch passed.
          -1 unit 91m 7s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          121m 53s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840009/HDFS-11156.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 60cd3d32f1a4 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a926f89
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17631/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17631/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17631/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17631/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 7s Maven dependency ordering for branch +1 mvninstall 8m 35s trunk passed +1 compile 1m 27s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 35s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 1m 7s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 1m 39s the patch passed +1 javac 1m 39s the patch passed -0 checkstyle 0m 34s hadoop-hdfs-project: The patch generated 1 new + 230 unchanged - 4 fixed = 231 total (was 234) +1 mvnsite 1m 41s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 23s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 0m 56s hadoop-hdfs-client in the patch passed. -1 unit 91m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 121m 53s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840009/HDFS-11156.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 60cd3d32f1a4 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a926f89 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17631/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17631/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17631/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17631/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 6s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 1m 20s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 3m 5s trunk passed
          +1 javadoc 0m 58s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 1m 17s the patch passed
          +1 javac 1m 17s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 3 new + 231 unchanged - 3 fixed = 234 total (was 234)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 53s the patch passed
          +1 unit 0m 53s hadoop-hdfs-client in the patch passed.
          +1 unit 61m 54s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          88m 9s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Private method org.apache.hadoop.hdfs.web.JsonUtil.toJsonArray(DatanodeInfo[]) is never called At JsonUtil.java:called At JsonUtil.java:[lines 189-198]
            Private method org.apache.hadoop.hdfs.web.JsonUtil.toJsonMap(ExtendedBlock) is never called At JsonUtil.java:called At JsonUtil.java:[lines 139-148]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11156
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839964/HDFS-11156.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 99268116dc01 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6f80742
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17630/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17630/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17630/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17630/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 6s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 1m 20s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 0m 58s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 1m 17s the patch passed +1 javac 1m 17s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 3 new + 231 unchanged - 3 fixed = 234 total (was 234) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 53s the patch passed +1 unit 0m 53s hadoop-hdfs-client in the patch passed. +1 unit 61m 54s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 88m 9s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Private method org.apache.hadoop.hdfs.web.JsonUtil.toJsonArray(DatanodeInfo[]) is never called At JsonUtil.java:called At JsonUtil.java: [lines 189-198]   Private method org.apache.hadoop.hdfs.web.JsonUtil.toJsonMap(ExtendedBlock) is never called At JsonUtil.java:called At JsonUtil.java: [lines 139-148] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11156 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839964/HDFS-11156.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 99268116dc01 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6f80742 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17630/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17630/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17630/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17630/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          +1 for the idea.

          I change the target version as 3.0+. Hope you don't mind.

          Show
          liuml07 Mingliang Liu added a comment - +1 for the idea. I change the target version as 3.0+. Hope you don't mind.

            People

            • Assignee:
              cheersyang Weiwei Yang
              Reporter:
              cheersyang Weiwei Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development