Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9799

Reimplement getCurrentTrashDir to remove incompatibility

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-8831 changed the signature of TrashPolicy#getCurrentTrashDir by adding an IOException. This breaks other applications using this public API. This JIRA aims to reimplement the logic to safely handle the IOException within HDFS.

      1. HDFS-9799.05.patch
        10 kB
        Zhe Zhang
      2. HDFS-9799.04.patch
        10 kB
        Zhe Zhang
      3. HDFS-9799.03.patch
        10 kB
        Zhe Zhang
      4. HDFS-9799.02.patch
        6 kB
        Zhe Zhang
      5. HDFS-9799.01.patch
        6 kB
        Zhe Zhang
      6. HDFS-9799.00.patch
        5 kB
        Zhe Zhang

        Issue Links

          Activity

          Hide
          zhz Zhe Zhang added a comment -

          The source of the IOException is from getEZForPath. So when getEZForPath gets an exception – meaning that the EZ of the given path cannot be determined at the time of calling, we should just return the Trash dir of the user's home. Even if the path does belong to an EZ, this will just mean the rm will fail later. With the added WARN message, the calling application should be able to obtain the RC of the rm failure.

          Andrew Wang Aaron T. Myers Could you take a look? Thanks.

          Show
          zhz Zhe Zhang added a comment - The source of the IOException is from getEZForPath . So when getEZForPath gets an exception – meaning that the EZ of the given path cannot be determined at the time of calling, we should just return the Trash dir of the user's home. Even if the path does belong to an EZ, this will just mean the rm will fail later. With the added WARN message, the calling application should be able to obtain the RC of the rm failure. Andrew Wang Aaron T. Myers Could you take a look? Thanks.
          Hide
          andrew.wang Andrew Wang added a comment -

          Throwing a new checked exception is definitely incompatible, good find.

          How about the getTrashRoots method that was added which also throws IOException? Seems like we should do a similar change for parity.

          Show
          andrew.wang Andrew Wang added a comment - Throwing a new checked exception is definitely incompatible, good find. How about the getTrashRoots method that was added which also throws IOException? Seems like we should do a similar change for parity.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Andrew for the review.

          The IOException logic in getTrashRoots is a little more complicated. Even the unencrypted case throws an exception caused by exists etc. If we swallow those exceptions we need to think about a meaningful return value (e.g., an empty collection?). Returning an empty collection also means we won't have detailed exception about which path caused issues.

          Show
          zhz Zhe Zhang added a comment - Thanks Andrew for the review. The IOException logic in getTrashRoots is a little more complicated. Even the unencrypted case throws an exception caused by exists etc. If we swallow those exceptions we need to think about a meaningful return value (e.g., an empty collection?). Returning an empty collection also means we won't have detailed exception about which path caused issues.
          Hide
          zhz Zhe Zhang added a comment -

          The v01 patch added some formatting and Javadoc changes.

          Show
          zhz Zhe Zhang added a comment - The v01 patch added some formatting and Javadoc changes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 15s Maven dependency ordering for branch
          +1 mvninstall 9m 20s trunk passed
          +1 compile 10m 56s trunk passed with JDK v1.8.0_72
          +1 compile 9m 43s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 2m 15s trunk passed
          +1 mvneclipse 0m 33s trunk passed
          +1 findbugs 4m 5s trunk passed
          +1 javadoc 1m 55s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 0s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 43s the patch passed
          +1 compile 11m 20s the patch passed with JDK v1.8.0_72
          +1 javac 11m 20s the patch passed
          +1 compile 9m 51s the patch passed with JDK v1.7.0_95
          +1 javac 9m 51s the patch passed
          -1 checkstyle 1m 27s root: patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191)
          +1 mvnsite 2m 8s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 41s the patch passed
          +1 javadoc 2m 2s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 3s the patch passed with JDK v1.7.0_95
          -1 unit 9m 51s hadoop-common in the patch failed with JDK v1.8.0_72.
          +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          -1 unit 11m 0s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 1m 25s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 28s Patch does not generate ASF License warnings.
          105m 43s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
          JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestIname



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787694/HDFS-9799.00.patch
          JIRA Issue HDFS-9799
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 38ed0bf7f172 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e6a7044
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14480/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14480/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 15s Maven dependency ordering for branch +1 mvninstall 9m 20s trunk passed +1 compile 10m 56s trunk passed with JDK v1.8.0_72 +1 compile 9m 43s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 28s trunk passed +1 mvnsite 2m 15s trunk passed +1 mvneclipse 0m 33s trunk passed +1 findbugs 4m 5s trunk passed +1 javadoc 1m 55s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 0s trunk passed with JDK v1.7.0_95 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 11m 20s the patch passed with JDK v1.8.0_72 +1 javac 11m 20s the patch passed +1 compile 9m 51s the patch passed with JDK v1.7.0_95 +1 javac 9m 51s the patch passed -1 checkstyle 1m 27s root: patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191) +1 mvnsite 2m 8s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 41s the patch passed +1 javadoc 2m 2s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 3s the patch passed with JDK v1.7.0_95 -1 unit 9m 51s hadoop-common in the patch failed with JDK v1.8.0_72. +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 11m 0s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 1m 25s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 105m 43s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestIname Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787694/HDFS-9799.00.patch JIRA Issue HDFS-9799 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 38ed0bf7f172 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e6a7044 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14480/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14480/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14480/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 6m 57s trunk passed with JDK v1.8.0_66
          +1 compile 7m 26s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 1m 43s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 26s trunk passed
          +1 javadoc 1m 20s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 34s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 6m 47s the patch passed with JDK v1.8.0_66
          +1 javac 6m 47s the patch passed
          +1 compile 7m 22s the patch passed with JDK v1.7.0_91
          +1 javac 7m 22s the patch passed
          -1 checkstyle 1m 11s root: patch generated 1 new + 190 unchanged - 1 fixed = 191 total (was 191)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 2s the patch passed
          +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 37s the patch passed with JDK v1.7.0_91
          -1 unit 8m 1s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 19s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          77m 40s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787704/HDFS-9799.01.patch
          JIRA Issue HDFS-9799
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2e947d9a6ac8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9fdfb54
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14481/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14481/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14481/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14481/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14481/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 6m 57s trunk passed with JDK v1.8.0_66 +1 compile 7m 26s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 9s trunk passed +1 mvnsite 1m 43s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 26s trunk passed +1 javadoc 1m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 34s trunk passed with JDK v1.7.0_91 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 6m 47s the patch passed with JDK v1.8.0_66 +1 javac 6m 47s the patch passed +1 compile 7m 22s the patch passed with JDK v1.7.0_91 +1 javac 7m 22s the patch passed -1 checkstyle 1m 11s root: patch generated 1 new + 190 unchanged - 1 fixed = 191 total (was 191) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 2s the patch passed +1 javadoc 1m 18s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 37s the patch passed with JDK v1.7.0_91 -1 unit 8m 1s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 8m 19s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 77m 40s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787704/HDFS-9799.01.patch JIRA Issue HDFS-9799 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2e947d9a6ac8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9fdfb54 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14481/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14481/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14481/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14481/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14481/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Updating the patch to fix checkstyle issue. The reported test failure is unrelated and passes locally.

          Show
          zhz Zhe Zhang added a comment - Updating the patch to fix checkstyle issue. The reported test failure is unrelated and passes locally.
          Hide
          andrew.wang Andrew Wang added a comment -

          From an API point of view, these two methods should be similar. Another flaw with getTrashRoots is that an exception for any single trash root fails the entire call. There's no ability to do a partial return, which IMO is better since it gives the caller flexibility.

          Regarding error handling, pre HDFS-8831 it looks like TrashPolicyDefault would just catch and log exceptions. So the exception itself isn't being used for much.

          Show
          andrew.wang Andrew Wang added a comment - From an API point of view, these two methods should be similar. Another flaw with getTrashRoots is that an exception for any single trash root fails the entire call. There's no ability to do a partial return, which IMO is better since it gives the caller flexibility. Regarding error handling, pre HDFS-8831 it looks like TrashPolicyDefault would just catch and log exceptions. So the exception itself isn't being used for much.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Andrew. I took another look at getTrashRoots. Agreed that returning the collection of trash root dirs before exception is safe and is more flexible. E.g. empitier can process those returned dirs. Attaching v03 patch to address.

          Show
          zhz Zhe Zhang added a comment - Thanks Andrew. I took another look at getTrashRoots . Agreed that returning the collection of trash root dirs before exception is safe and is more flexible. E.g. empitier can process those returned dirs. Attaching v03 patch to address.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 7m 8s trunk passed
          +1 compile 6m 35s trunk passed with JDK v1.8.0_66
          +1 compile 7m 16s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 1m 41s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 29s trunk passed
          +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 32s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 6m 40s the patch passed with JDK v1.8.0_66
          +1 javac 6m 40s the patch passed
          +1 compile 7m 24s the patch passed with JDK v1.7.0_91
          +1 javac 7m 24s the patch passed
          +1 checkstyle 1m 9s root: patch generated 0 new + 190 unchanged - 1 fixed = 190 total (was 191)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 3s the patch passed
          +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 31s the patch passed with JDK v1.7.0_91
          +1 unit 8m 19s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 21s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          77m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787728/HDFS-9799.02.patch
          JIRA Issue HDFS-9799
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 11f08424eac8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1de1641
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14485/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14485/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 8s trunk passed +1 compile 6m 35s trunk passed with JDK v1.8.0_66 +1 compile 7m 16s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 10s trunk passed +1 mvnsite 1m 41s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 29s trunk passed +1 javadoc 1m 19s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 32s trunk passed with JDK v1.7.0_91 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 6m 40s the patch passed with JDK v1.8.0_66 +1 javac 6m 40s the patch passed +1 compile 7m 24s the patch passed with JDK v1.7.0_91 +1 javac 7m 24s the patch passed +1 checkstyle 1m 9s root: patch generated 0 new + 190 unchanged - 1 fixed = 190 total (was 191) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 3s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 31s the patch passed with JDK v1.7.0_91 +1 unit 8m 19s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 8m 21s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 77m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787728/HDFS-9799.02.patch JIRA Issue HDFS-9799 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11f08424eac8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1de1641 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14485/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14485/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 31s trunk passed
          +1 compile 6m 31s trunk passed with JDK v1.8.0_72
          +1 compile 6m 51s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 7s trunk passed
          +1 mvnsite 1m 37s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 7s trunk passed
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 28s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 5m 38s the patch passed with JDK v1.8.0_72
          +1 javac 5m 38s the patch passed
          +1 compile 6m 35s the patch passed with JDK v1.7.0_95
          +1 javac 6m 35s the patch passed
          -1 checkstyle 1m 42s root: patch generated 2 new + 190 unchanged - 1 fixed = 192 total (was 191)
          +1 mvnsite 1m 34s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 37s the patch passed
          +1 javadoc 1m 13s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 26s the patch passed with JDK v1.7.0_95
          +1 unit 7m 25s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          72m 46s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787745/HDFS-9799.03.patch
          JIRA Issue HDFS-9799
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dfe066e66099 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1de1641
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14487/artifact/patchprocess/diff-checkstyle-root.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14487/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14487/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 31s trunk passed +1 compile 6m 31s trunk passed with JDK v1.8.0_72 +1 compile 6m 51s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 7s trunk passed +1 mvnsite 1m 37s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 7s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 28s trunk passed with JDK v1.7.0_95 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 5m 38s the patch passed with JDK v1.8.0_72 +1 javac 5m 38s the patch passed +1 compile 6m 35s the patch passed with JDK v1.7.0_95 +1 javac 6m 35s the patch passed -1 checkstyle 1m 42s root: patch generated 2 new + 190 unchanged - 1 fixed = 192 total (was 191) +1 mvnsite 1m 34s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 37s the patch passed +1 javadoc 1m 13s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 26s the patch passed with JDK v1.7.0_95 +1 unit 7m 25s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 72m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787745/HDFS-9799.03.patch JIRA Issue HDFS-9799 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dfe066e66099 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1de1641 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14487/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14487/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14487/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Fixing checkstyle issues (2 long lines, trivial change)

          Show
          zhz Zhe Zhang added a comment - Fixing checkstyle issues (2 long lines, trivial change)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 7m 5s trunk passed
          +1 compile 6m 49s trunk passed with JDK v1.8.0_66
          +1 compile 7m 13s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 1m 42s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 27s trunk passed
          +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 33s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 6m 36s the patch passed with JDK v1.8.0_66
          +1 javac 6m 36s the patch passed
          +1 compile 7m 11s the patch passed with JDK v1.7.0_91
          +1 javac 7m 11s the patch passed
          +1 checkstyle 1m 8s root: patch generated 0 new + 190 unchanged - 1 fixed = 190 total (was 191)
          +1 mvnsite 1m 41s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 48s the patch passed
          +1 javadoc 1m 21s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 33s the patch passed with JDK v1.7.0_91
          +1 unit 8m 16s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          76m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787767/HDFS-9799.04.patch
          JIRA Issue HDFS-9799
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 11d808975d37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 91a96ea
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14489/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14489/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 7m 5s trunk passed +1 compile 6m 49s trunk passed with JDK v1.8.0_66 +1 compile 7m 13s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 10s trunk passed +1 mvnsite 1m 42s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 27s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 33s trunk passed with JDK v1.7.0_91 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 6m 36s the patch passed with JDK v1.8.0_66 +1 javac 6m 36s the patch passed +1 compile 7m 11s the patch passed with JDK v1.7.0_91 +1 javac 7m 11s the patch passed +1 checkstyle 1m 8s root: patch generated 0 new + 190 unchanged - 1 fixed = 190 total (was 191) +1 mvnsite 1m 41s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 48s the patch passed +1 javadoc 1m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 33s the patch passed with JDK v1.7.0_91 +1 unit 8m 16s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 76m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787767/HDFS-9799.04.patch JIRA Issue HDFS-9799 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11d808975d37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 91a96ea Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14489/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14489/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Zhe Zhang for reporting the issue and working on the fix.

          The source of the IOException is from getEZForPath. So when getEZForPath gets an exception – meaning that the EZ of the given path cannot be determined at the time of calling, we should just return the Trash dir of the user's home. Even if the path does belong to an EZ, this will just mean the rm will fail later.

          Can you elaborate when getEZForPath gets an IOException? Based on EncryptionZoneManager#getEZINodeForPath, getEZForPath() just returns null instead of throwing IOException when a given path cannot be determined to be inside a EZ. This makes DFS#getTrashRoots() to include just the Trash dir of the user's home as returned result for non-EZ paths. Should we just fix the annotations?

          Show
          xyao Xiaoyu Yao added a comment - Thanks Zhe Zhang for reporting the issue and working on the fix. The source of the IOException is from getEZForPath. So when getEZForPath gets an exception – meaning that the EZ of the given path cannot be determined at the time of calling, we should just return the Trash dir of the user's home. Even if the path does belong to an EZ, this will just mean the rm will fail later. Can you elaborate when getEZForPath gets an IOException? Based on EncryptionZoneManager#getEZINodeForPath, getEZForPath() just returns null instead of throwing IOException when a given path cannot be determined to be inside a EZ. This makes DFS#getTrashRoots() to include just the Trash dir of the user's home as returned result for non-EZ paths. Should we just fix the annotations?
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiaoyu Yao for the discussion.

          FSNamesystem#getEZForPath gets IOExceptions at multiple places:

          1. checkOperation
          2. FSDirEncryptionZoneOp#getEZForPath => resolvePath, getINodesInPath etc.

          So the IOException from the getEZForPath call itself is pretty hard to get rid of. We could try swallowing and handling the IOException at DFSClient#getEZForPath level. But removing an exception from a public API will cause surprises to applications as well.

          So I suggest we handle the exception at getTrashRoot / getTrashRoots level. Let me know if this makes sense.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiaoyu Yao for the discussion. FSNamesystem#getEZForPath gets IOExceptions at multiple places: checkOperation FSDirEncryptionZoneOp#getEZForPath => resolvePath , getINodesInPath etc. So the IOException from the getEZForPath call itself is pretty hard to get rid of. We could try swallowing and handling the IOException at DFSClient#getEZForPath level. But removing an exception from a public API will cause surprises to applications as well. So I suggest we handle the exception at getTrashRoot / getTrashRoots level. Let me know if this makes sense.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Zhe Zhang for the explanation. Agree with your changes on getTrashRoot()/getTrashRoots level. For the change in getTrashRoots, can we add some indication for partial results being returned at API level in addition to the log.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Zhe Zhang for the explanation. Agree with your changes on getTrashRoot()/getTrashRoots level. For the change in getTrashRoots, can we add some indication for partial results being returned at API level in addition to the log.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the suggestion Xiaoyu Yao. An API level indication would be a nice. I explored 2 approaches:

          1. Add null elements in the returned collection
          2. Create a new data structure, something like GetTrashRootsResult, with a boolean flag indicating partial results

          With either approach, it's pretty complex to reflect the structure of returned trash roots. This is because both FileSystem and DistributedFileSystem levels could return partial results (and in principle, any subclass of FileSystem could return partial results too). Given the complexity I suggest we just rely on WARN logging. Please see if you agree.

          Updating the patch with minor cosmetic changes.

          Show
          zhz Zhe Zhang added a comment - Thanks for the suggestion Xiaoyu Yao . An API level indication would be a nice. I explored 2 approaches: Add null elements in the returned collection Create a new data structure, something like GetTrashRootsResult , with a boolean flag indicating partial results With either approach, it's pretty complex to reflect the structure of returned trash roots. This is because both FileSystem and DistributedFileSystem levels could return partial results (and in principle, any subclass of FileSystem could return partial results too). Given the complexity I suggest we just rely on WARN logging. Please see if you agree. Updating the patch with minor cosmetic changes.
          Hide
          andrew.wang Andrew Wang added a comment -

          The ideal way of doing partial results is something like Optional, though that's Java 8 and still won't work perfectly since if we fail to list, we won't know the number of homedirs/EZs. It's something we could consider for trunk, if we really care about it.

          IMO the current idea is good. The most likely cause of IOExceptions is if something throws an AccessControlException if it's not being run as a superuser, and the logging should help with that.

          One little nit though is that we aren't printing the exception in these logs, which would help with debugging:

          LOG.warn("Cannot get all trash roots.");
          ...
          DFSClient.LOG.warn("Cannot get all encrypted trash roots.");
          

          Otherwise I'm +1, thanks Zhe.

          Show
          andrew.wang Andrew Wang added a comment - The ideal way of doing partial results is something like Optional, though that's Java 8 and still won't work perfectly since if we fail to list, we won't know the number of homedirs/EZs. It's something we could consider for trunk, if we really care about it. IMO the current idea is good. The most likely cause of IOExceptions is if something throws an AccessControlException if it's not being run as a superuser, and the logging should help with that. One little nit though is that we aren't printing the exception in these logs, which would help with debugging: LOG.warn( "Cannot get all trash roots." ); ... DFSClient.LOG.warn( "Cannot get all encrypted trash roots." ); Otherwise I'm +1, thanks Zhe.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 48s Maven dependency ordering for branch
          +1 mvninstall 6m 48s trunk passed
          +1 compile 5m 43s trunk passed with JDK v1.8.0_72
          +1 compile 6m 28s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 7s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 8s trunk passed
          +1 javadoc 1m 10s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 27s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 10s the patch passed
          +1 compile 6m 4s the patch passed with JDK v1.8.0_72
          +1 javac 6m 4s the patch passed
          +1 compile 6m 35s the patch passed with JDK v1.7.0_95
          +1 javac 6m 35s the patch passed
          +1 checkstyle 1m 6s root: patch generated 0 new + 190 unchanged - 1 fixed = 190 total (was 191)
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 37s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 25s the patch passed with JDK v1.7.0_95
          +1 unit 7m 4s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 21s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          70m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788294/HDFS-9799.05.patch
          JIRA Issue HDFS-9799
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 399dad32b494 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3a23dc6
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14514/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14514/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 6m 48s trunk passed +1 compile 5m 43s trunk passed with JDK v1.8.0_72 +1 compile 6m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 7s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 8s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 27s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 10s the patch passed +1 compile 6m 4s the patch passed with JDK v1.8.0_72 +1 javac 6m 4s the patch passed +1 compile 6m 35s the patch passed with JDK v1.7.0_95 +1 javac 6m 35s the patch passed +1 checkstyle 1m 6s root: patch generated 0 new + 190 unchanged - 1 fixed = 190 total (was 191) +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 37s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 25s the patch passed with JDK v1.7.0_95 +1 unit 7m 4s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. +1 unit 7m 21s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 70m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788294/HDFS-9799.05.patch JIRA Issue HDFS-9799 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 399dad32b494 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3a23dc6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14514/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14514/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Andrew for the final review! I just committed the patch to trunk, branch-2 and branch-2.8.

          Thanks Xiaoyu and Arpit for the helpful comments. If we see a further need to improve partial result returning I can file a follow-on JIRA.

          Show
          zhz Zhe Zhang added a comment - Thanks Andrew for the final review! I just committed the patch to trunk, branch-2 and branch-2.8. Thanks Xiaoyu and Arpit for the helpful comments. If we see a further need to improve partial result returning I can file a follow-on JIRA.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9316 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9316/)
          HDFS-9799. Reimplement getCurrentTrashDir to remove incompatibility. (zhz: rev 0fb14aaced7afdbbb72e59316c186ebf6ec2d091)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicy.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9316 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9316/ ) HDFS-9799 . Reimplement getCurrentTrashDir to remove incompatibility. (zhz: rev 0fb14aaced7afdbbb72e59316c186ebf6ec2d091) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicy.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java

            People

            • Assignee:
              zhz Zhe Zhang
              Reporter:
              zhz Zhe Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development