Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10756

Expose getTrashRoot to HTTPFS and WebHDFS

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      "getTrashRoot" returns a trash root for a path. Currently in DFS if the path "/foo" is a normal path, it returns "/user/$USER/.Trash" for "/foo" and if "/foo" is an encrypted zone, it returns "/foo/.Trash/$USER" for the child file/dir of "/foo". This patch is about to override the old "getTrashRoot" of httpfs and webhdfs, so that the behavior of returning trash root in httpfs and webhdfs are consistent with DFS.
      Show
      "getTrashRoot" returns a trash root for a path. Currently in DFS if the path "/foo" is a normal path, it returns "/user/$USER/.Trash" for "/foo" and if "/foo" is an encrypted zone, it returns "/foo/.Trash/$USER" for the child file/dir of "/foo". This patch is about to override the old "getTrashRoot" of httpfs and webhdfs, so that the behavior of returning trash root in httpfs and webhdfs are consistent with DFS.

      Description

      Currently, hadoop FileSystem API has getTrashRoot to determine trash directory at run time. Default trash dir is under /user/$USER

      For an encrypted file, since moving files between/in/out of EZs are not allowed, when an EZ file is deleted via CLI, it calls in to DFS implementation to move the file to a trash directory under the same EZ.

      This works perfectly fine for CLI users or java users who call FileSystem API. But for users via httpfs/webhdfs, currently there is no way to figure out what the trash root would be. This jira is proposing we add such interface to httpfs and webhdfs.

      1. HDFS-10756.001.patch
        28 kB
        Yuanbo Liu
      2. HDFS-10756.002.patch
        32 kB
        Yuanbo Liu
      3. HDFS-10756.003.patch
        24 kB
        Yuanbo Liu
      4. HDFS-10756.004.patch
        28 kB
        Yuanbo Liu
      5. HDFS-10756.005.patch
        32 kB
        Yuanbo Liu
      6. HDFS-10756.006.patch
        31 kB
        Yuanbo Liu
      7. HDFS-10756.007.patch
        31 kB
        Yuanbo Liu

        Issue Links

          Activity

          Hide
          yuanbo Yuanbo Liu added a comment -

          Xiao Chen
          I've tested EZ deletion in my own environment, and my hadoop version is 2.7.2.
          Actually I find that EZ file can not be moved from EZ to a trash directory under the same EZ while using hadoop client.
          I do some research about this issue and find a comment from Harsh, see the details from here: https://community.cloudera.com/t5/Storage-Random-Access-HDFS/Is-it-possible-to-use-trash-in-HDFS-encryption-zone/td-p/36567
          He suggests that we create a .Trash in the EZ directory and use "mv" not "rm" to delete the file in EZ.
          If we want to use "rm" command to delete EZ file, we have to add option "-skipTrash" which apparently is not related to getTrashRoot
          Is my understanding right?

          Show
          yuanbo Yuanbo Liu added a comment - Xiao Chen I've tested EZ deletion in my own environment, and my hadoop version is 2.7.2. Actually I find that EZ file can not be moved from EZ to a trash directory under the same EZ while using hadoop client. I do some research about this issue and find a comment from Harsh, see the details from here: https://community.cloudera.com/t5/Storage-Random-Access-HDFS/Is-it-possible-to-use-trash-in-HDFS-encryption-zone/td-p/36567 He suggests that we create a .Trash in the EZ directory and use "mv" not "rm" to delete the file in EZ. If we want to use "rm" command to delete EZ file, we have to add option "-skipTrash" which apparently is not related to getTrashRoot Is my understanding right?
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Yuanbo Liu You're absolutely correct. The trash support for EZ is added after (unreleased) Hadoop 2.8.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Yuanbo Liu You're absolutely correct. The trash support for EZ is added after (unreleased) Hadoop 2.8.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Yuanbo Liu for looking into this.

          Actually I find that EZ file can not be moved from EZ to a trash directory under the same EZ while using hadoop client.

          Could you give an example of what you mean by this?
          From CLI, hdfs dfs -rm /ez/file will rename (mv) the file to a trash directory (/ez1/.Trash/$USER/file, which is decided using getTrashRoot), and hdfs dfs -rm -skipTrash /ez/file will permanently delete the file.

          The issue I'm reporting here is that, for a webhdfs/httpfs client (imagine a Python script accessing hdfs), since the rest API does not support trash, a file can only be deleted permanently. Without EZ, the client can workaround this by rename the file into /user/username/.Trash. But EZ restricts such rename operation, and it has to be moved to /ez1/.Trash/$USER, which the client has no way to know. So my proposal here is to add getTrashRoot so webhdfs/httpfs clients know where to rename the file to, without worrying about the file being in a EZ or not.

          An alternate way is to maybe add a -moveToTrash to the delete api on webhdfs/httpfs, but I'm not sure whether that's feasible.

          Show
          xiaochen Xiao Chen added a comment - Thanks Yuanbo Liu for looking into this. Actually I find that EZ file can not be moved from EZ to a trash directory under the same EZ while using hadoop client. Could you give an example of what you mean by this? From CLI, hdfs dfs -rm /ez/file will rename ( mv ) the file to a trash directory (/ez1/.Trash/$USER/file, which is decided using getTrashRoot), and hdfs dfs -rm -skipTrash /ez/file will permanently delete the file. The issue I'm reporting here is that, for a webhdfs/httpfs client (imagine a Python script accessing hdfs), since the rest API does not support trash, a file can only be deleted permanently. Without EZ, the client can workaround this by rename the file into /user/username/.Trash . But EZ restricts such rename operation, and it has to be moved to /ez1/.Trash/$USER , which the client has no way to know. So my proposal here is to add getTrashRoot so webhdfs/httpfs clients know where to rename the file to, without worrying about the file being in a EZ or not. An alternate way is to maybe add a -moveToTrash to the delete api on webhdfs/httpfs, but I'm not sure whether that's feasible.
          Hide
          xiaochen Xiao Chen added a comment -

          Ah, I see. Thanks Wei-Chiu for closing the gaps.

          Show
          xiaochen Xiao Chen added a comment - Ah, I see. Thanks Wei-Chiu for closing the gaps.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I guess Yuanbo Liu's question is: do we want to support trash in HttpFSFileSystem and WebHDFSFileSystem, or support skip-trash-delete, or both.

          Show
          jojochuang Wei-Chiu Chuang added a comment - I guess Yuanbo Liu 's question is: do we want to support trash in HttpFSFileSystem and WebHDFSFileSystem, or support skip-trash-delete, or both.
          Hide
          xiaochen Xiao Chen added a comment -

          Delete without trash already works.
          IMO ideally we can add trash to httpfs/webhdfs, if that's doable (obviously can't be default for compatibility reasons). We should expose getTrashRoot at a minimum. I think the above two can be in parallel.

          Show
          xiaochen Xiao Chen added a comment - Delete without trash already works. IMO ideally we can add trash to httpfs/webhdfs, if that's doable (obviously can't be default for compatibility reasons). We should expose getTrashRoot at a minimum. I think the above two can be in parallel.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Xiao Chen Wei-Chiu Chuang Thanks for your response!
          I see why I can't use EZ trash, I thought hadoop did not support EZ trash. I choose the wrong branch.

          I guess Yuanbo Liu's question is: do we

          Since EZ deletion has support Trash, supporting both in HttpFSFileSystem and WebHDFSFileSystem seems more reasonable.

          We should expose getTrashRoot at a minimum.

          Yes I agree, I will test it later and figure out what I can do.
          Thank you both for your time and comments !

          Show
          yuanbo Yuanbo Liu added a comment - Xiao Chen Wei-Chiu Chuang Thanks for your response! I see why I can't use EZ trash, I thought hadoop did not support EZ trash. I choose the wrong branch. I guess Yuanbo Liu's question is: do we Since EZ deletion has support Trash, supporting both in HttpFSFileSystem and WebHDFSFileSystem seems more reasonable. We should expose getTrashRoot at a minimum. Yes I agree, I will test it later and figure out what I can do. Thank you both for your time and comments !
          Hide
          yuanbo Yuanbo Liu added a comment -

          Uploaded v1 patch for this issue.
          It would be a bit big patch, since we need to expose getTrashRoot both in HTTPFS and WebHDFS. How about split this jira into three sub-tasks.
          1) expose getTrashRoot in WebHDFS
          2) expose getTrashRoot in HTTPFS
          3) document it in webHDFS.md
          By the way, the patch v1 does not contain document change. I'd like to get your thoughts before further action.
          Thanks for your time!

          Show
          yuanbo Yuanbo Liu added a comment - Uploaded v1 patch for this issue. It would be a bit big patch, since we need to expose getTrashRoot both in HTTPFS and WebHDFS . How about split this jira into three sub-tasks. 1) expose getTrashRoot in WebHDFS 2) expose getTrashRoot in HTTPFS 3) document it in webHDFS.md By the way, the patch v1 does not contain document change. I'd like to get your thoughts before further action. Thanks for your time!
          Hide
          xiaochen Xiao Chen added a comment -

          Thank you for working on this, Yuanbo Liu. Idea looks good.

          I don't have a strong feeling regarding splitting the jira. If we're splitting though, I'd prefer we have just 2 jiras for webhdfs and httpfs respectively, and both include their own doc changes.

          Some comments:

          • In both implementations' getTrashRoot:
            • I see we return null when an IOE is caught. This is not the same behavior as dfs, which always returns super.getTrashRoot(path);.
            • LOG.warn("can not find trash root of " + p);: Since we're using slf4j, let's use place holders when logging. Also, suggest s/can not/Cannot/g here.
          • FSOperations.java
            • Suggest we keep the whitespace as-is, to minimize our change.
            • @InterfaceAudience.Private should be on a separate line on its own.
          • NamenodeWebHdfsMethods.java
            • getTrashRoot's inner comment is not accurate. queryPath.isRoot() == true doesn't mean it is not an EZ. If / is an EZ, we choose to user's home dir because it's under / as well. How about we add a javadoc to the method, and remove the comment here?
          • Test
            • Can we add a case to verify nested ez's trash dir works too?
          • Please fix the checkstyles when jenkins comes back.
          Show
          xiaochen Xiao Chen added a comment - Thank you for working on this, Yuanbo Liu . Idea looks good. I don't have a strong feeling regarding splitting the jira. If we're splitting though, I'd prefer we have just 2 jiras for webhdfs and httpfs respectively, and both include their own doc changes. Some comments: In both implementations' getTrashRoot : I see we return null when an IOE is caught. This is not the same behavior as dfs, which always returns super.getTrashRoot(path); . LOG.warn("can not find trash root of " + p); : Since we're using slf4j, let's use place holders when logging. Also, suggest s/can not/Cannot/g here. FSOperations.java Suggest we keep the whitespace as-is, to minimize our change. @InterfaceAudience.Private should be on a separate line on its own. NamenodeWebHdfsMethods.java getTrashRoot 's inner comment is not accurate. queryPath.isRoot() == true doesn't mean it is not an EZ. If / is an EZ, we choose to user's home dir because it's under / as well. How about we add a javadoc to the method, and remove the comment here? Test Can we add a case to verify nested ez's trash dir works too? Please fix the checkstyles when jenkins comes back.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 1m 37s Maven dependency ordering for branch
          +1 mvninstall 8m 12s trunk passed
          +1 compile 8m 32s trunk passed
          +1 checkstyle 1m 59s trunk passed
          +1 mvnsite 3m 23s trunk passed
          +1 mvneclipse 0m 57s trunk passed
          +1 findbugs 5m 27s trunk passed
          +1 javadoc 2m 33s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 3m 1s the patch passed
          +1 compile 8m 45s the patch passed
          -1 javac 8m 45s root generated 1 new + 708 unchanged - 1 fixed = 709 total (was 709)
          -0 checkstyle 1m 57s root: The patch generated 3 new + 941 unchanged - 0 fixed = 944 total (was 941)
          +1 mvnsite 3m 27s the patch passed
          +1 mvneclipse 1m 5s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 6m 6s the patch passed
          +1 javadoc 2m 42s the patch passed
          +1 unit 9m 13s hadoop-common in the patch passed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 76m 47s hadoop-hdfs in the patch failed.
          -1 unit 3m 37s hadoop-hdfs-httpfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          152m 48s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestRollingUpgrade
            hadoop.fs.http.server.TestHttpFSServer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824723/HDFS-10756.001.patch
          JIRA Issue HDFS-10756
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b7efb030e82c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c49333b
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16504/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16504/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 1m 37s Maven dependency ordering for branch +1 mvninstall 8m 12s trunk passed +1 compile 8m 32s trunk passed +1 checkstyle 1m 59s trunk passed +1 mvnsite 3m 23s trunk passed +1 mvneclipse 0m 57s trunk passed +1 findbugs 5m 27s trunk passed +1 javadoc 2m 33s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 3m 1s the patch passed +1 compile 8m 45s the patch passed -1 javac 8m 45s root generated 1 new + 708 unchanged - 1 fixed = 709 total (was 709) -0 checkstyle 1m 57s root: The patch generated 3 new + 941 unchanged - 0 fixed = 944 total (was 941) +1 mvnsite 3m 27s the patch passed +1 mvneclipse 1m 5s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 6s the patch passed +1 javadoc 2m 42s the patch passed +1 unit 9m 13s hadoop-common in the patch passed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 76m 47s hadoop-hdfs in the patch failed. -1 unit 3m 37s hadoop-hdfs-httpfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 152m 48s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestRollingUpgrade   hadoop.fs.http.server.TestHttpFSServer Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824723/HDFS-10756.001.patch JIRA Issue HDFS-10756 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b7efb030e82c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c49333b Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16504/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16504/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16504/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hello Yuanbo Liu, nice work!
          On top of Xiao's comment, I have a few quick comments:

          • when logging an exception, please make sure to log the exception itself. The exception itself (including the log message and the stacktrace) is a useful information when debugging.
          • can you also verify the behavior of getTrashRoot() with a non-EZ path?

          Also, I would like to point out that for any downstream applications that use the proposed GETTRASHROOT operation, it is important to note that the application should not assume the trash directory returned exists. Downstream applications should have some way of handling this, and create trash directory with appropriate sticky bit and permission. See HDFS-10324 for discussion.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hello Yuanbo Liu , nice work! On top of Xiao's comment, I have a few quick comments: when logging an exception, please make sure to log the exception itself. The exception itself (including the log message and the stacktrace) is a useful information when debugging. can you also verify the behavior of getTrashRoot() with a non-EZ path? Also, I would like to point out that for any downstream applications that use the proposed GETTRASHROOT operation, it is important to note that the application should not assume the trash directory returned exists. Downstream applications should have some way of handling this, and create trash directory with appropriate sticky bit and permission. See HDFS-10324 for discussion.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Wei-Chiu Chuang and Xiao Chen Thanks a lot for your comments.
          I will read through your suggestions and response them later.
          Thanks again for your time!

          Show
          yuanbo Yuanbo Liu added a comment - Wei-Chiu Chuang and Xiao Chen Thanks a lot for your comments. I will read through your suggestions and response them later. Thanks again for your time!
          Hide
          yuanbo Yuanbo Liu added a comment - - edited

          Wei-Chiu Chuang / Xiao Chen Sorry for the late response, I'm quite busy last week.

          1. From Xiaochen

          I don't have a strong feeling regarding splitting the jira.

          I thought this patch might be too large to review. I'm OK if we use a single jira to track it.

          In both implementations'......

          Good catch! I've changed my code in V2 patch. One thing I should high line is that I didn't write EZ test case in BaseTestHttpFSWith. I've reviewed the code in BaseTestHttpFSWith, it would be tricky if I add EZ test case here. So I added such kind of code in TestHttpFSServer.

          I also change the code in DistributedFileSystem#getTrashRoot, here is the change

          -    if ((path == null) || path.isRoot() || !dfs.isHDFSEncryptionEnabled()) {
          +    if ((path == null) || path.isRoot()) {
                 return super.getTrashRoot(path);
               }
          -    String parentSrc = path.getParent().toUri().getPath();
          +    String pathSrc = path.toUri().getPath();
          

          Because the configuration of client may not be up-to-date, it's not accurate to use client's isHDFSEncryptionEnabled to detect whether EZ is enabled.
          Also using the parent path to get EZ for the path is not right in some cases. For example, if the EZ path is "/zone", the parent path "/" is a normal path, getTrashRoot returns wrong info about "/zone".

          2. From Weichiu

          when logging an exception..

          Good suggestions! I think my v2 patch has addressed them.

          it is important to note that the application should not assume the trash directory returned exists

          First I thought returning null in getTrashRoot may help to remind users that the trash path may not exist. But Xiaochen reminded me that returning null is not the same behavior in DFS's getTrashRoot. So I wrote some test cases personally and found that

          • If the input path is a normal path, the result of trash root is related to the user who invoke getTrashRoot
          • If the input path is a EZ path, the result of trash root is related to the EZ path and the user.
          • If exception happens int item 2), it falls back to the default implement aka item 1) and returns "/user/$USER/.Trash"
          • The result of trash root is not necessarily existing in HDFS.
            In case of the behaviors of DFS, I didn't write code to inform users that the trash path may not exist.

          Thanks for your time and I hope to get your thoughts.

          Show
          yuanbo Yuanbo Liu added a comment - - edited Wei-Chiu Chuang / Xiao Chen Sorry for the late response, I'm quite busy last week. 1. From Xiaochen I don't have a strong feeling regarding splitting the jira. I thought this patch might be too large to review. I'm OK if we use a single jira to track it. In both implementations'...... Good catch! I've changed my code in V2 patch. One thing I should high line is that I didn't write EZ test case in BaseTestHttpFSWith . I've reviewed the code in BaseTestHttpFSWith , it would be tricky if I add EZ test case here. So I added such kind of code in TestHttpFSServer . I also change the code in DistributedFileSystem#getTrashRoot , here is the change - if ((path == null ) || path.isRoot() || !dfs.isHDFSEncryptionEnabled()) { + if ((path == null ) || path.isRoot()) { return super .getTrashRoot(path); } - String parentSrc = path.getParent().toUri().getPath(); + String pathSrc = path.toUri().getPath(); Because the configuration of client may not be up-to-date, it's not accurate to use client's isHDFSEncryptionEnabled to detect whether EZ is enabled. Also using the parent path to get EZ for the path is not right in some cases. For example, if the EZ path is "/zone", the parent path "/" is a normal path, getTrashRoot returns wrong info about "/zone". 2. From Weichiu when logging an exception.. Good suggestions! I think my v2 patch has addressed them. it is important to note that the application should not assume the trash directory returned exists First I thought returning null in getTrashRoot may help to remind users that the trash path may not exist. But Xiaochen reminded me that returning null is not the same behavior in DFS's getTrashRoot. So I wrote some test cases personally and found that If the input path is a normal path, the result of trash root is related to the user who invoke getTrashRoot If the input path is a EZ path, the result of trash root is related to the EZ path and the user. If exception happens int item 2), it falls back to the default implement aka item 1) and returns "/user/$USER/.Trash" The result of trash root is not necessarily existing in HDFS. In case of the behaviors of DFS, I didn't write code to inform users that the trash path may not exist. Thanks for your time and I hope to get your thoughts.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 45s trunk passed
          +1 compile 8m 15s trunk passed
          +1 checkstyle 1m 42s trunk passed
          +1 mvnsite 3m 15s trunk passed
          +1 mvneclipse 0m 56s trunk passed
          +1 findbugs 5m 52s trunk passed
          +1 javadoc 2m 26s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 30s the patch passed
          +1 compile 7m 18s the patch passed
          +1 javac 7m 18s the patch passed
          +1 checkstyle 1m 37s root: The patch generated 0 new + 959 unchanged - 1 fixed = 959 total (was 960)
          +1 mvnsite 2m 48s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 30s the patch passed
          +1 javadoc 2m 15s the patch passed
          +1 unit 7m 45s hadoop-common in the patch passed.
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed.
          -1 unit 59m 45s hadoop-hdfs in the patch failed.
          +1 unit 3m 24s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          127m 7s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestEncryptionZonesWithKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10756
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826147/HDFS-10756.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9bd0bcc2996e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4bd45f5
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16574/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16574/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16574/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 45s trunk passed +1 compile 8m 15s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 3m 15s trunk passed +1 mvneclipse 0m 56s trunk passed +1 findbugs 5m 52s trunk passed +1 javadoc 2m 26s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 30s the patch passed +1 compile 7m 18s the patch passed +1 javac 7m 18s the patch passed +1 checkstyle 1m 37s root: The patch generated 0 new + 959 unchanged - 1 fixed = 959 total (was 960) +1 mvnsite 2m 48s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 30s the patch passed +1 javadoc 2m 15s the patch passed +1 unit 7m 45s hadoop-common in the patch passed. +1 unit 0m 55s hadoop-hdfs-client in the patch passed. -1 unit 59m 45s hadoop-hdfs in the patch failed. +1 unit 3m 24s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 127m 7s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestEncryptionZonesWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826147/HDFS-10756.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9bd0bcc2996e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4bd45f5 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16574/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16574/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16574/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Yuanbo Liu for the new patch! Overall looks good. Some comments from me:

          • The change in DistributedFileSystem#getTrashRoot.
            Could you explain more? Looking into DFSUtilClient#isHDFSEncryptionEnabled, it's checking the provider URIs. Without provider URIs the client can't contact KMS server, and hence won't be able to get the key. I think the check is correct.
          • NamenodeWebHdfsMethods
            Please stick with current coding style. Needs '{}' to wrap the case block.
          • ... BaseTestHttpFSWith, it would be tricky if I add EZ test case here
            Could you explain what the difficulty is? I think Andrew's HDFS-10832 may be a good example on how to add the tests into the base class.
          • Failed tests look related, likely due to the parent path change in DFS.java
          • Could you also add a couple more test cases to cover getTrash when: / as an EZ, nested EZ?
          Show
          xiaochen Xiao Chen added a comment - Thanks Yuanbo Liu for the new patch! Overall looks good. Some comments from me: The change in DistributedFileSystem#getTrashRoot . Could you explain more? Looking into DFSUtilClient#isHDFSEncryptionEnabled , it's checking the provider URIs. Without provider URIs the client can't contact KMS server, and hence won't be able to get the key. I think the check is correct. NamenodeWebHdfsMethods Please stick with current coding style. Needs '{}' to wrap the case block. ... BaseTestHttpFSWith, it would be tricky if I add EZ test case here Could you explain what the difficulty is? I think Andrew's HDFS-10832 may be a good example on how to add the tests into the base class. Failed tests look related, likely due to the parent path change in DFS.java Could you also add a couple more test cases to cover getTrash when: / as an EZ, nested EZ?
          Hide
          yuanbo Yuanbo Liu added a comment -

          Xiao Chen Thanks for your comments.
          We have a holiday next week(mid-autumn day), and I will take a vacation, so I will reply it later. Thanks again for your time!

          Show
          yuanbo Yuanbo Liu added a comment - Xiao Chen Thanks for your comments. We have a holiday next week(mid-autumn day), and I will take a vacation, so I will reply it later. Thanks again for your time!
          Hide
          xiaochen Xiao Chen added a comment -

          We have a holiday next week(mid-autumn day)

          Sure, thanks for the heads-up. Wish you enjoy some good mooncakes with your family.

          Also forgot to say in my last comment, we should add docs in this patch (to WebHDFS.md for example).

          Show
          xiaochen Xiao Chen added a comment - We have a holiday next week(mid-autumn day) Sure, thanks for the heads-up. Wish you enjoy some good mooncakes with your family. Also forgot to say in my last comment, we should add docs in this patch (to WebHDFS.md for example).
          Hide
          yuanbo Yuanbo Liu added a comment -

          I'm working on a related jira( HDFS-10883 ), and I will focus on this jira after I fix that.

          Show
          yuanbo Yuanbo Liu added a comment - I'm working on a related jira( HDFS-10883 ), and I will focus on this jira after I fix that.
          Hide
          yuanbo Yuanbo Liu added a comment - - edited

          Xiao Chen Sorry for the late response.
          Since HDFS-10883 has been done, I provide v3 patch for this issue according to your last comment and my v2 patch.
          Some statement:
          1) Adding EZ test case of WebHDFS in TestEncryptionZones
          2) Adding EZ test case of HttpFS in TestHttpFSServer instead of BaseTestHttpFSWith, it's much more easier to test EZ there.
          3) Getting rid of some check style bugs in BaseTestHttpFSWith. It's annoying to see those bugs .

          Show
          yuanbo Yuanbo Liu added a comment - - edited Xiao Chen Sorry for the late response. Since HDFS-10883 has been done, I provide v3 patch for this issue according to your last comment and my v2 patch. Some statement: 1) Adding EZ test case of WebHDFS in TestEncryptionZones 2) Adding EZ test case of HttpFS in TestHttpFSServer instead of BaseTestHttpFSWith , it's much more easier to test EZ there. 3) Getting rid of some check style bugs in BaseTestHttpFSWith . It's annoying to see those bugs .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 1m 34s Maven dependency ordering for branch
          +1 mvninstall 7m 29s trunk passed
          +1 compile 7m 35s trunk passed
          +1 checkstyle 1m 41s trunk passed
          +1 mvnsite 3m 2s trunk passed
          +1 mvneclipse 0m 51s trunk passed
          +1 findbugs 5m 14s trunk passed
          +1 javadoc 1m 51s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 21s the patch passed
          +1 compile 7m 2s the patch passed
          +1 javac 7m 2s the patch passed
          -0 checkstyle 1m 38s root: The patch generated 1 new + 871 unchanged - 76 fixed = 872 total (was 947)
          +1 mvnsite 2m 48s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 47s the patch passed
          +1 javadoc 1m 57s the patch passed
          +1 unit 7m 49s hadoop-common in the patch passed.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 6s hadoop-hdfs in the patch failed.
          +1 unit 3m 49s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          130m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10756
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834944/HDFS-10756.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f8578e6495b8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b18f35f
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17263/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17263/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17263/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17263/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 1m 34s Maven dependency ordering for branch +1 mvninstall 7m 29s trunk passed +1 compile 7m 35s trunk passed +1 checkstyle 1m 41s trunk passed +1 mvnsite 3m 2s trunk passed +1 mvneclipse 0m 51s trunk passed +1 findbugs 5m 14s trunk passed +1 javadoc 1m 51s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 21s the patch passed +1 compile 7m 2s the patch passed +1 javac 7m 2s the patch passed -0 checkstyle 1m 38s root: The patch generated 1 new + 871 unchanged - 76 fixed = 872 total (was 947) +1 mvnsite 2m 48s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 47s the patch passed +1 javadoc 1m 57s the patch passed +1 unit 7m 49s hadoop-common in the patch passed. +1 unit 1m 2s hadoop-hdfs-client in the patch passed. -1 unit 64m 6s hadoop-hdfs in the patch failed. +1 unit 3m 49s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 130m 28s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeLifeline Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834944/HDFS-10756.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f8578e6495b8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b18f35f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17263/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17263/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17263/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17263/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          The test failure is not related to my code change, it's a known failure.Get more details from here: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/200/

          The checkstyle failure is not worthy of fixing. It's not a consistent behavior to add white space before ".run()"

          Show
          yuanbo Yuanbo Liu added a comment - The test failure is not related to my code change, it's a known failure.Get more details from here: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/200/ The checkstyle failure is not worthy of fixing. It's not a consistent behavior to add white space before ".run()"
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for revving. Patch looks good.

          Some comments:

          • Let's add coverage in TestEncryptionZones for nested EZ. I see / is covered now, which is great.
          • HttpFSFileSystem.java: Map<String, String> params = new HashMap<String, String>(); can leave out the type and instantiate the HashMap with a diamond operator <>

          Also, some of my earlier comments still need to be addressed:

          • NamenodeWebHdfsMethods
            Please stick with current coding style. Needs '{}' to wrap the case block.
          • we should add docs in this patch (to WebHDFS.md for example).
          Show
          xiaochen Xiao Chen added a comment - Thanks for revving. Patch looks good. Some comments: Let's add coverage in TestEncryptionZones for nested EZ. I see / is covered now, which is great. HttpFSFileSystem.java : Map<String, String> params = new HashMap<String, String>(); can leave out the type and instantiate the HashMap with a diamond operator <> Also, some of my earlier comments still need to be addressed: NamenodeWebHdfsMethods Please stick with current coding style. Needs '{}' to wrap the case block. we should add docs in this patch (to WebHDFS.md for example).
          Hide
          yuanbo Yuanbo Liu added a comment -

          Xiao Chen Thanks for you response.
          upload v4 patch to address your comment.

          Show
          yuanbo Yuanbo Liu added a comment - Xiao Chen Thanks for you response. upload v4 patch to address your comment.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 23s Maven dependency ordering for branch
          +1 mvninstall 8m 52s trunk passed
          +1 compile 8m 58s trunk passed
          +1 checkstyle 1m 59s trunk passed
          +1 mvnsite 3m 28s trunk passed
          +1 mvneclipse 0m 58s trunk passed
          +1 findbugs 5m 33s trunk passed
          +1 javadoc 2m 7s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 44s the patch passed
          +1 compile 7m 59s the patch passed
          +1 javac 7m 59s the patch passed
          -0 checkstyle 1m 35s root: The patch generated 3 new + 871 unchanged - 76 fixed = 874 total (was 947)
          +1 mvnsite 2m 42s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 17s the patch passed
          +1 javadoc 1m 53s the patch passed
          +1 unit 8m 36s hadoop-common in the patch passed.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed.
          -1 unit 63m 21s hadoop-hdfs in the patch failed.
          +1 unit 3m 49s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          134m 18s



          Reason Tests
          Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestFileAppend4
            org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            org.apache.hadoop.hdfs.server.namenode.TestNameNodeReconfigure
            org.apache.hadoop.fs.TestFcHdfsCreateMkdir
            org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
            org.apache.hadoop.hdfs.server.mover.TestStorageMover
            org.apache.hadoop.hdfs.TestRestartDFS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10756
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835314/HDFS-10756.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3227880f413f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 24a83fe
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17290/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17290/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17290/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17290/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 8m 52s trunk passed +1 compile 8m 58s trunk passed +1 checkstyle 1m 59s trunk passed +1 mvnsite 3m 28s trunk passed +1 mvneclipse 0m 58s trunk passed +1 findbugs 5m 33s trunk passed +1 javadoc 2m 7s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 44s the patch passed +1 compile 7m 59s the patch passed +1 javac 7m 59s the patch passed -0 checkstyle 1m 35s root: The patch generated 3 new + 871 unchanged - 76 fixed = 874 total (was 947) +1 mvnsite 2m 42s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 17s the patch passed +1 javadoc 1m 53s the patch passed +1 unit 8m 36s hadoop-common in the patch passed. +1 unit 1m 1s hadoop-hdfs-client in the patch passed. -1 unit 63m 21s hadoop-hdfs in the patch failed. +1 unit 3m 49s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 134m 18s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestFileAppend4   org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   org.apache.hadoop.hdfs.server.namenode.TestNameNodeReconfigure   org.apache.hadoop.fs.TestFcHdfsCreateMkdir   org.apache.hadoop.hdfs.server.namenode.TestFileTruncate   org.apache.hadoop.hdfs.server.mover.TestStorageMover   org.apache.hadoop.hdfs.TestRestartDFS Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835314/HDFS-10756.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3227880f413f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 24a83fe Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17290/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17290/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17290/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17290/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Those test cases fail because of "Java heap space", I don't think it's related to my code change.

          Show
          yuanbo Yuanbo Liu added a comment - Those test cases fail because of "Java heap space", I don't think it's related to my code change.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Yuanbo Liu for the new rev. Some comments, mostly nits:

          HttpFSFileSystem:

          • Would prefer to have a new TRASH_ROOT_JSON instead of mixing with HOME_DIR_JSON. And update both sides to use this.

          In the documentation

          • Hoping we can add something like javadoc from DFS to explain this overall.
          • Suggest to add a link to TransparentEncryption.html#Rename_and_Trash_considerations as well, to help explain the trash-over-EZ logic.
          • if the path is an encrypted zone path: s/if/If/g, s/is an encrypted zone path/is in an encryption zone/g.
          • and user has permission to the path
            I believe the javadoc of DFS will take care of this. Also ping Andrew Wang for his view regarding this: if user has no perm, we return default as DFS does. While this is consistent and has a warn log, I find it a somewhat surprising behavior. Any advice (as-is or throw or sth else)? Sorry I should have asked when I propose the change earlier.
          • In the example "Path": "/user/szetszwo/.Trash", let's use username, or a more general name like alice/bob, as TransparentEncryption.html did.

          BaseTestHttpFSWith

          • pretty trivial, but foo is a less aggressive word than fool. I suggest we use the former as var names.

          TestHttpFSServer.java:

          • extra line after getPath method, inside and after testGetTrashRoot method. Basically please use 1 line break.
          • it's very thorough to test GETTRASHROOT with delegation tokens. But need to connect and assert after having the url.
          Show
          xiaochen Xiao Chen added a comment - Thanks Yuanbo Liu for the new rev. Some comments, mostly nits: HttpFSFileSystem: Would prefer to have a new TRASH_ROOT_JSON instead of mixing with HOME_DIR_JSON . And update both sides to use this. In the documentation Hoping we can add something like javadoc from DFS to explain this overall. Suggest to add a link to TransparentEncryption.html#Rename_and_Trash_considerations as well, to help explain the trash-over-EZ logic. if the path is an encrypted zone path : s/if/If/g, s/is an encrypted zone path/is in an encryption zone/g. and user has permission to the path I believe the javadoc of DFS will take care of this. Also ping Andrew Wang for his view regarding this: if user has no perm, we return default as DFS does. While this is consistent and has a warn log, I find it a somewhat surprising behavior. Any advice (as-is or throw or sth else)? Sorry I should have asked when I propose the change earlier. In the example "Path": "/user/szetszwo/.Trash" , let's use username , or a more general name like alice/bob, as TransparentEncryption.html did. BaseTestHttpFSWith pretty trivial, but foo is a less aggressive word than fool . I suggest we use the former as var names. TestHttpFSServer.java: extra line after getPath method, inside and after testGetTrashRoot method. Basically please use 1 line break. it's very thorough to test GETTRASHROOT with delegation tokens. But need to connect and assert after having the url.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Xiao Chen Thanks for your comments.
          upload v5 patch to address most of your comments.

          While this is consistent and has a warn log, I find it a somewhat surprising behavior. ..

          Agree with you that it's a wired behavior. These two ideas get into my mind if we want to fix it:
          1. Return null if there is a running error
          2. Create a new run time exception and throw it.
          I didn't change the behavior in my v5 patch. Since many filesystem's classes need to be modified and verified if the origin DFS's behavior changes, and it will be good to file another JIRA to track such issue, agree?

          Show
          yuanbo Yuanbo Liu added a comment - Xiao Chen Thanks for your comments. upload v5 patch to address most of your comments. While this is consistent and has a warn log, I find it a somewhat surprising behavior. .. Agree with you that it's a wired behavior. These two ideas get into my mind if we want to fix it: 1. Return null if there is a running error 2. Create a new run time exception and throw it. I didn't change the behavior in my v5 patch. Since many filesystem's classes need to be modified and verified if the origin DFS's behavior changes, and it will be good to file another JIRA to track such issue, agree?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 1m 46s Maven dependency ordering for branch
          +1 mvninstall 6m 46s trunk passed
          +1 compile 6m 52s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 2m 48s trunk passed
          +1 mvneclipse 0m 50s trunk passed
          +1 findbugs 4m 49s trunk passed
          +1 javadoc 1m 57s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 20s the patch passed
          +1 compile 6m 49s the patch passed
          +1 javac 6m 49s the patch passed
          -0 checkstyle 1m 36s root: The patch generated 5 new + 889 unchanged - 76 fixed = 894 total (was 965)
          +1 mvnsite 2m 44s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 33s the patch passed
          +1 javadoc 2m 5s the patch passed
          -1 unit 8m 2s hadoop-common in the patch failed.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          +1 unit 59m 22s hadoop-hdfs in the patch passed.
          +1 unit 3m 18s hadoop-hdfs-httpfs in the patch passed.
          -1 asflicense 16m 49s The patch generated 4 ASF License warnings.
          139m 38s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10756
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835749/HDFS-10756.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d49e342c5bcb 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8fa2195
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17339/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17339/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17339/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17339/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17339/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 1m 46s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 6m 52s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 2m 48s trunk passed +1 mvneclipse 0m 50s trunk passed +1 findbugs 4m 49s trunk passed +1 javadoc 1m 57s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 20s the patch passed +1 compile 6m 49s the patch passed +1 javac 6m 49s the patch passed -0 checkstyle 1m 36s root: The patch generated 5 new + 889 unchanged - 76 fixed = 894 total (was 965) +1 mvnsite 2m 44s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 33s the patch passed +1 javadoc 2m 5s the patch passed -1 unit 8m 2s hadoop-common in the patch failed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. +1 unit 59m 22s hadoop-hdfs in the patch passed. +1 unit 3m 18s hadoop-hdfs-httpfs in the patch passed. -1 asflicense 16m 49s The patch generated 4 ASF License warnings. 139m 38s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835749/HDFS-10756.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d49e342c5bcb 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8fa2195 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17339/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17339/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17339/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17339/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17339/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thank you for the new patch, Yuanbo Liu.

          Hoping we can add something like javadoc from DFS to explain this overall.... add a link to TransparentEncryption.html#Rename_and_Trash_considerations

          Sorry for any miscommunication. These comments are for the WebHDFS.md documentation. No need to update current javadoc from DFS, but please add similar text to the WebHDFS.md documentation, and also add a link to TransparentEncryption.html#Rename_and_Trash_considerations so interested user can find more details.
          Please update the HttpFSFileSystem javadoc accordingly.

          ...it's a wired behavior ... file another JIRA to track such issue

          I agree it shouldn't be changed here. Changing that would be incompatible. I think that worths a discussion first: It appears the fall-back logic was added due to HDFS-9799, to solve an incompatible behavior in branch-2. But wondering if we should modify 3.0 to a more intuitive behavior. Zhe Zhang / Andrew Wang / other watchers, thoughts?

          Show
          xiaochen Xiao Chen added a comment - Thank you for the new patch, Yuanbo Liu . Hoping we can add something like javadoc from DFS to explain this overall.... add a link to TransparentEncryption.html#Rename_and_Trash_considerations Sorry for any miscommunication. These comments are for the WebHDFS.md documentation. No need to update current javadoc from DFS, but please add similar text to the WebHDFS.md documentation, and also add a link to TransparentEncryption.html#Rename_and_Trash_considerations so interested user can find more details. Please update the HttpFSFileSystem javadoc accordingly. ...it's a wired behavior ... file another JIRA to track such issue I agree it shouldn't be changed here. Changing that would be incompatible. I think that worths a discussion first: It appears the fall-back logic was added due to HDFS-9799 , to solve an incompatible behavior in branch-2. But wondering if we should modify 3.0 to a more intuitive behavior. Zhe Zhang / Andrew Wang / other watchers, thoughts?
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the ping, and sorry I missed the earlier one. Yea, I think we should file a new JIRA to track changing this, as it is incompatible. I will say that the FileSystem API is very sticky, so my first thought is that we add a new API that throws exceptions rather than changing the existing one.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the ping, and sorry I missed the earlier one. Yea, I think we should file a new JIRA to track changing this, as it is incompatible. I will say that the FileSystem API is very sticky, so my first thought is that we add a new API that throws exceptions rather than changing the existing one.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Xiao Chen Thanks for your comments. Upload v6 patch.
          1. Revert the code comment in DistributedFileSystem#getTrashRoot
          2. Update WebHDFS.md and add the EZ link.
          3. Update index.md of HttpFS about GETTRASHROOT operation.
          4. Change the description of code comment in HttpFSFileSystem#getTrashRoot.
          Hope my v6 patch can address your comments.

          Show
          yuanbo Yuanbo Liu added a comment - Xiao Chen Thanks for your comments. Upload v6 patch. 1. Revert the code comment in DistributedFileSystem#getTrashRoot 2. Update WebHDFS.md and add the EZ link. 3. Update index.md of HttpFS about GETTRASHROOT operation. 4. Change the description of code comment in HttpFSFileSystem#getTrashRoot . Hope my v6 patch can address your comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 8m 35s trunk passed
          +1 compile 8m 49s trunk passed
          +1 checkstyle 1m 53s trunk passed
          +1 mvnsite 3m 27s trunk passed
          +1 mvneclipse 0m 55s trunk passed
          +1 findbugs 5m 21s trunk passed
          +1 javadoc 1m 56s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 23s the patch passed
          +1 compile 6m 45s the patch passed
          +1 javac 6m 45s the patch passed
          -0 checkstyle 1m 38s root: The patch generated 3 new + 871 unchanged - 76 fixed = 874 total (was 947)
          +1 mvnsite 2m 43s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 39s the patch passed
          +1 javadoc 2m 1s the patch passed
          +1 unit 8m 0s hadoop-common in the patch passed.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed.
          +1 unit 59m 14s hadoop-hdfs in the patch passed.
          +1 unit 3m 17s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          126m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10756
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836268/HDFS-10756.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2b06fe3d2cdd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7ba74be
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17364/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17364/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17364/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 8m 35s trunk passed +1 compile 8m 49s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 3m 27s trunk passed +1 mvneclipse 0m 55s trunk passed +1 findbugs 5m 21s trunk passed +1 javadoc 1m 56s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 23s the patch passed +1 compile 6m 45s the patch passed +1 javac 6m 45s the patch passed -0 checkstyle 1m 38s root: The patch generated 3 new + 871 unchanged - 76 fixed = 874 total (was 947) +1 mvnsite 2m 43s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 39s the patch passed +1 javadoc 2m 1s the patch passed +1 unit 8m 0s hadoop-common in the patch passed. +1 unit 1m 0s hadoop-hdfs-client in the patch passed. +1 unit 59m 14s hadoop-hdfs in the patch passed. +1 unit 3m 17s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 126m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836268/HDFS-10756.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2b06fe3d2cdd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ba74be Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17364/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17364/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17364/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the new version, Yuanbo Liu. Some additional comments:

          • The WebHDFS doc isn't correctly formatted. Please mvn site and double check. Namely, 'For more details about trash root in an encrypted zone, please refer to Transparent Encryption Guide' need to be on a new line. Also need a . in the end of the sentence.
          • WebHdfsFileSystem: prefer to just call getUri instead of this.getUri.

          Otherwise looks good.
          Also thanks Andrew Wang for the comment, I just created HDFS-11091 for that.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the new version, Yuanbo Liu . Some additional comments: The WebHDFS doc isn't correctly formatted. Please mvn site and double check. Namely, 'For more details about trash root in an encrypted zone, please refer to Transparent Encryption Guide' need to be on a new line. Also need a . in the end of the sentence. WebHdfsFileSystem: prefer to just call getUri instead of this.getUri . Otherwise looks good. Also thanks Andrew Wang for the comment, I just created HDFS-11091 for that.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Xiao Chen
          Sorry for the mistake link, upload v7 patch.
          If you don't mind, I will take HDFS-11091 over to see what I can do.

          Show
          yuanbo Yuanbo Liu added a comment - Xiao Chen Sorry for the mistake link, upload v7 patch. If you don't mind, I will take HDFS-11091 over to see what I can do.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 1m 35s Maven dependency ordering for branch
          +1 mvninstall 8m 15s trunk passed
          +1 compile 8m 12s trunk passed
          +1 checkstyle 1m 40s trunk passed
          +1 mvnsite 3m 4s trunk passed
          +1 mvneclipse 0m 56s trunk passed
          +1 findbugs 5m 21s trunk passed
          +1 javadoc 2m 0s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 34s the patch passed
          +1 compile 8m 11s the patch passed
          +1 javac 8m 11s the patch passed
          -0 checkstyle 1m 41s root: The patch generated 3 new + 870 unchanged - 76 fixed = 873 total (was 946)
          +1 mvnsite 3m 19s the patch passed
          +1 mvneclipse 0m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 6m 19s the patch passed
          +1 javadoc 2m 9s the patch passed
          +1 unit 8m 45s hadoop-common in the patch passed.
          +1 unit 1m 5s hadoop-hdfs-client in the patch passed.
          -1 unit 61m 59s hadoop-hdfs in the patch failed.
          +1 unit 3m 33s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          133m 38s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestEncryptionZones



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10756
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836515/HDFS-10756.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d45f952efcfe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cb5cc0d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17379/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17379/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17379/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17379/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 1m 35s Maven dependency ordering for branch +1 mvninstall 8m 15s trunk passed +1 compile 8m 12s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 3m 4s trunk passed +1 mvneclipse 0m 56s trunk passed +1 findbugs 5m 21s trunk passed +1 javadoc 2m 0s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 34s the patch passed +1 compile 8m 11s the patch passed +1 javac 8m 11s the patch passed -0 checkstyle 1m 41s root: The patch generated 3 new + 870 unchanged - 76 fixed = 873 total (was 946) +1 mvnsite 3m 19s the patch passed +1 mvneclipse 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 6m 19s the patch passed +1 javadoc 2m 9s the patch passed +1 unit 8m 45s hadoop-common in the patch passed. +1 unit 1m 5s hadoop-hdfs-client in the patch passed. -1 unit 61m 59s hadoop-hdfs in the patch failed. +1 unit 3m 33s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 133m 38s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptionZones Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836515/HDFS-10756.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d45f952efcfe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cb5cc0d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17379/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17379/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17379/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17379/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Yuanbo, patch 7 LGTM. Won't fix the checkstyles to keep consistency with current code base. The test failure seems unrelated, and created HDFS-11093 for that.

          Will wait until Friday to commit, in case Wei-Chiu Chuang, Andrew Wang or other watchers have comments.

          If you don't mind, I will take HDFS-11091 over to see what I can do.

          Sure, thanks for asking.

          Show
          xiaochen Xiao Chen added a comment - Thanks Yuanbo, patch 7 LGTM. Won't fix the checkstyles to keep consistency with current code base. The test failure seems unrelated, and created HDFS-11093 for that. Will wait until Friday to commit, in case Wei-Chiu Chuang , Andrew Wang or other watchers have comments. If you don't mind, I will take HDFS-11091 over to see what I can do. Sure, thanks for asking.
          Hide
          xiaochen Xiao Chen added a comment -

          +1 to patch 7, will commit by end of today if no objections.

          Show
          xiaochen Xiao Chen added a comment - +1 to patch 7, will commit by end of today if no objections.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10777 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10777/)
          HDFS-10756. Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by (xiao: rev d8bab3dcb693b2773ede9a6e4f71ae85ee056f79)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/index.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10777 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10777/ ) HDFS-10756 . Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by (xiao: rev d8bab3dcb693b2773ede9a6e4f71ae85ee056f79) (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOpsCountStatistics.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/resources/GetOpParam.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/index.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java
          Hide
          xiaochen Xiao Chen added a comment -

          Committed this to trunk and branch-2.
          Thanks Yuanbo for the work, Wei-Chiu and Andrew for the reviews.

          Hi Yuanbo Liu, could you post a short release note? Thanks.

          Show
          xiaochen Xiao Chen added a comment - Committed this to trunk and branch-2. Thanks Yuanbo for the work, Wei-Chiu and Andrew for the reviews. Hi Yuanbo Liu , could you post a short release note? Thanks.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Sure, I've updated the release note.

          Show
          yuanbo Yuanbo Liu added a comment - Sure, I've updated the release note.

            People

            • Assignee:
              yuanbo Yuanbo Liu
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development