Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8831

Trash Support for deletion in HDFS encryption zone

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: encryption
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Add Trash support for deleting files within encryption zones. Deleted files will remain encrypted and they will be moved to a “.Trash” subdirectory under the root of the encryption zone, prefixed by $USER/current. Checkpoint and expunge continue to work like the existing Trash.
      Show
      Add Trash support for deleting files within encryption zones. Deleted files will remain encrypted and they will be moved to a “.Trash” subdirectory under the root of the encryption zone, prefixed by $USER/current. Checkpoint and expunge continue to work like the existing Trash.

      Description

      Currently, "Soft Delete" is only supported if the whole encryption zone is deleted. If you delete files whinin the zone with trash feature enabled, you will get error similar to the following

      rm: Failed to move to trash: hdfs://HW11217.local:9000/z1_1/startnn.sh: /z1_1/startnn.sh can't be moved from an encryption zone.
      

      This JIRA is proposed to support trash for deletion of files within encryption zone.

      1. HDFS-8831.00.patch
        25 kB
        Xiaoyu Yao
      2. HDFS-8831.01.patch
        25 kB
        Xiaoyu Yao
      3. HDFS-8831.02.patch
        26 kB
        Xiaoyu Yao
      4. HDFS-8831.03.patch
        27 kB
        Xiaoyu Yao
      5. HDFS-8831.04.patch
        27 kB
        Xiaoyu Yao
      6. HDFS-8831.05.patch
        28 kB
        Xiaoyu Yao
      7. HDFS-8831-10152015.pdf
        218 kB
        Xiaoyu Yao

        Issue Links

          Activity

          Hide
          xyao Xiaoyu Yao added a comment -

          Propose to add a .Trash under the root of encryption zone without dependency on HDFS-8830. This per encryption zone .Trash directory lives within the zone and can be created at the time of the encryption zone creation. Files deleted within the encryption zone will be renamed/checkpointed to .Trash directory inside the encryption zone. For example, user foo delete a file /ez1/bar from an encryption zone rooted at /ez1, the file will be move to trash at /ez1/.Trash/foo/Current/ez1/bar. This way, we remove the dependency on HDFS-8830 and avoid the management overhead because we don't need to explicitly add trash locations (current and checkpoints) outside to the encryption zone root for each users' current and checkpoint directories. When the entire encryption zone is deleted, it should work as it is today, i.e., the root of the encryption zone is moved to the trash folder of the user who executed the deletion.

          I will upload a proposal shortly that compares this with other alternatives for encryption zone trash support and summarize why the one proposed above is preferred. Any feedback and thoughts?

          Show
          xyao Xiaoyu Yao added a comment - Propose to add a .Trash under the root of encryption zone without dependency on HDFS-8830 . This per encryption zone .Trash directory lives within the zone and can be created at the time of the encryption zone creation. Files deleted within the encryption zone will be renamed/checkpointed to .Trash directory inside the encryption zone. For example, user foo delete a file /ez1/bar from an encryption zone rooted at /ez1, the file will be move to trash at /ez1/.Trash/foo/Current/ez1/bar. This way, we remove the dependency on HDFS-8830 and avoid the management overhead because we don't need to explicitly add trash locations (current and checkpoints) outside to the encryption zone root for each users' current and checkpoint directories. When the entire encryption zone is deleted, it should work as it is today, i.e., the root of the encryption zone is moved to the trash folder of the user who executed the deletion. I will upload a proposal shortly that compares this with other alternatives for encryption zone trash support and summarize why the one proposed above is preferred. Any feedback and thoughts?
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a design document for review and discussion.

          Show
          xyao Xiaoyu Yao added a comment - Attach a design document for review and discussion.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for working on this. The design doc is very helpful.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for working on this. The design doc is very helpful.
          Hide
          xyao Xiaoyu Yao added a comment -

          Attach an initial patch for Jenkins.

          Show
          xyao Xiaoyu Yao added a comment - Attach an initial patch for Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 44s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          -1 javac 8m 1s The applied patch generated 2 additional warning messages.
          +1 javadoc 10m 31s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 45s The applied patch generated 32 new checkstyle issues (total was 182, now 211).
          -1 whitespace 0m 3s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 6m 33s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 43s Tests failed in hadoop-common.
          -1 hdfs tests 52m 3s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 38s Tests passed in hadoop-hdfs-client.
              113m 18s  



          Reason Tests
          FindBugs module:hadoop-common
          FindBugs module:hadoop-hdfs-client
          Failed unit tests hadoop.fs.TestHarFileSystem
            hadoop.fs.TestTrash
            hadoop.fs.TestFilterFileSystem
            hadoop.fs.viewfs.TestViewFsTrash
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.TestHDFSTrash



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767833/HDFS-8831.00.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 25f8f80
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13115/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13115/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 44s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. -1 javac 8m 1s The applied patch generated 2 additional warning messages. +1 javadoc 10m 31s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 45s The applied patch generated 32 new checkstyle issues (total was 182, now 211). -1 whitespace 0m 3s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 6m 33s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. -1 common tests 7m 43s Tests failed in hadoop-common. -1 hdfs tests 52m 3s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 38s Tests passed in hadoop-hdfs-client.     113m 18s   Reason Tests FindBugs module:hadoop-common FindBugs module:hadoop-hdfs-client Failed unit tests hadoop.fs.TestHarFileSystem   hadoop.fs.TestTrash   hadoop.fs.TestFilterFileSystem   hadoop.fs.viewfs.TestViewFsTrash   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.TestHDFSTrash Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767833/HDFS-8831.00.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 25f8f80 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13115/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13115/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13115/console This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Fix unit test failures and trash issues with viewfs.

          Show
          xyao Xiaoyu Yao added a comment - Fix unit test failures and trash issues with viewfs.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 15s Findbugs (version 3.0.0) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          -1 javac 8m 6s The applied patch generated 2 additional warning messages.
          +1 javadoc 10m 31s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 10s The applied patch generated 5 new checkstyle issues (total was 199, now 201).
          -1 whitespace 0m 3s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 7m 4s The patch appears to introduce 4 new Findbugs (version 3.0.0) warnings.
          -1 common tests 6m 41s Tests failed in hadoop-common.
          -1 hdfs tests 54m 58s Tests failed in hadoop-hdfs.
          -1 hdfs tests 0m 40s Tests failed in hadoop-hdfs-client.
              113m 44s  



          Reason Tests
          FindBugs module:hadoop-common
          FindBugs module:hadoop-hdfs
          FindBugs module:hadoop-hdfs-client
          Failed unit tests hadoop.metrics2.sink.TestFileSink
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.TestEncryptionZones
          Failed build hadoop-hdfs-client



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768402/HDFS-8831.01.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 15eb84b
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13170/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13170/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 15s Findbugs (version 3.0.0) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. -1 javac 8m 6s The applied patch generated 2 additional warning messages. +1 javadoc 10m 31s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 10s The applied patch generated 5 new checkstyle issues (total was 199, now 201). -1 whitespace 0m 3s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 7m 4s The patch appears to introduce 4 new Findbugs (version 3.0.0) warnings. -1 common tests 6m 41s Tests failed in hadoop-common. -1 hdfs tests 54m 58s Tests failed in hadoop-hdfs. -1 hdfs tests 0m 40s Tests failed in hadoop-hdfs-client.     113m 44s   Reason Tests FindBugs module:hadoop-common FindBugs module:hadoop-hdfs FindBugs module:hadoop-hdfs-client Failed unit tests hadoop.metrics2.sink.TestFileSink   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.TestEncryptionZones Failed build hadoop-hdfs-client Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768402/HDFS-8831.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 15eb84b javac https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13170/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13170/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13170/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 22m 16s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          -1 javac 8m 58s The applied patch generated 2 additional warning messages.
          +1 javadoc 12m 36s There were no new javadoc warning messages.
          +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 46s The applied patch generated 5 new checkstyle issues (total was 199, now 201).
          -1 whitespace 0m 4s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 52s mvn install still works.
          +1 eclipse:eclipse 0m 42s The patch built with eclipse:eclipse.
          -1 findbugs 9m 39s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
          -1 common tests 7m 33s Tests failed in hadoop-common.
          -1 hdfs tests 54m 33s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 32s Tests passed in hadoop-hdfs-client.
              123m 1s  



          Reason Tests
          FindBugs module:hadoop-common
          FindBugs module:hadoop-hdfs-client
          Failed unit tests hadoop.ipc.TestDecayRpcScheduler
            hadoop.net.TestDNS
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.util.TestByteArrayManager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768402/HDFS-8831.01.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 15eb84b
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/diffcheckstylehadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13173/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13173/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 22m 16s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. -1 javac 8m 58s The applied patch generated 2 additional warning messages. +1 javadoc 12m 36s There were no new javadoc warning messages. +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 46s The applied patch generated 5 new checkstyle issues (total was 199, now 201). -1 whitespace 0m 4s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 52s mvn install still works. +1 eclipse:eclipse 0m 42s The patch built with eclipse:eclipse. -1 findbugs 9m 39s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. -1 common tests 7m 33s Tests failed in hadoop-common. -1 hdfs tests 54m 33s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 32s Tests passed in hadoop-hdfs-client.     123m 1s   Reason Tests FindBugs module:hadoop-common FindBugs module:hadoop-hdfs-client Failed unit tests hadoop.ipc.TestDecayRpcScheduler   hadoop.net.TestDNS   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.util.TestByteArrayManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768402/HDFS-8831.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 15eb84b Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13173/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13173/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13173/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 24m 2s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          -1 javac 8m 31s The applied patch generated 2 additional warning messages.
          +1 javadoc 11m 5s There were no new javadoc warning messages.
          +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 49s The applied patch generated 3 new checkstyle issues (total was 200, now 200).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 59s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          -1 findbugs 7m 35s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
          -1 common tests 9m 1s Tests failed in hadoop-common.
          -1 hdfs tests 66m 30s Tests failed in hadoop-hdfs.
          +1 hdfs tests 0m 36s Tests passed in hadoop-hdfs-client.
              134m 11s  



          Reason Tests
          FindBugs module:hadoop-common
          FindBugs module:hadoop-hdfs-client
          Failed unit tests hadoop.fs.TestLocalFsFCStatistics
            hadoop.ipc.TestDecayRpcScheduler
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.blockmanagement.TestNodeCount



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768496/HDFS-8831.02.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7781fe1
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/testrun_hadoop-hdfs.txt
          hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13179/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13179/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 24m 2s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. -1 javac 8m 31s The applied patch generated 2 additional warning messages. +1 javadoc 11m 5s There were no new javadoc warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 49s The applied patch generated 3 new checkstyle issues (total was 200, now 200). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 59s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. -1 findbugs 7m 35s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. -1 common tests 9m 1s Tests failed in hadoop-common. -1 hdfs tests 66m 30s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 36s Tests passed in hadoop-hdfs-client.     134m 11s   Reason Tests FindBugs module:hadoop-common FindBugs module:hadoop-hdfs-client Failed unit tests hadoop.fs.TestLocalFsFCStatistics   hadoop.ipc.TestDecayRpcScheduler   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.blockmanagement.TestNodeCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768496/HDFS-8831.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7781fe1 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/13179/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13179/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13179/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Xiaoyu Yao, thanks for the detailed design note. My comments, mostly around potential compatibility of classes tagged @InterfaceAudience.Public.

          1. DistributedFileSystem.java:2326: We can skip the call to dfs.getEZForPath if isHDFSEncryptionEnabled is false to avoid extra RPC call when TDE is not enabled.
          2. FileSystem.java:2701: Can we define .Trash as a constant somewhere?
          3. Trash.java:98: Avoid extra RPC for log statement. Can we cache the currentTrashDir some time earlier?
          4. TrashPolicy.java:48: I don't think we should mark it as deprecated. While the TrashPolicyDefault no longer uses the home parameter other implementations may be passing a different value here in theory.
          5. TrashPolicy.java:57: Also we should have a default implementation of this routine else it will be a backward incompatible change (will break existing implementations of this public interface).
          6. TrashPolicy.java:83: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller.
          7. TrashPolicy.java:92: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller.
          8. TrashPolicy.java:108: We should leave the old method in place to keep the public interface backwards compatible. Perhaps to be conservative we should respect the 'home' parameter if one is passed in instead of using Filesystem#getTrashRoot?

          https://github.com/arp7/hadoop/commit/7b3212d2c41cc35cce81eadc68c029e0fc67a429

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Xiaoyu Yao , thanks for the detailed design note. My comments, mostly around potential compatibility of classes tagged @InterfaceAudience.Public . DistributedFileSystem.java:2326: We can skip the call to dfs.getEZForPath if isHDFSEncryptionEnabled is false to avoid extra RPC call when TDE is not enabled. FileSystem.java:2701: Can we define .Trash as a constant somewhere? Trash.java:98: Avoid extra RPC for log statement. Can we cache the currentTrashDir some time earlier? TrashPolicy.java:48: I don't think we should mark it as deprecated. While the TrashPolicyDefault no longer uses the home parameter other implementations may be passing a different value here in theory. TrashPolicy.java:57: Also we should have a default implementation of this routine else it will be a backward incompatible change (will break existing implementations of this public interface). TrashPolicy.java:83: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller. TrashPolicy.java:92: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller. TrashPolicy.java:108: We should leave the old method in place to keep the public interface backwards compatible. Perhaps to be conservative we should respect the 'home' parameter if one is passed in instead of using Filesystem#getTrashRoot? https://github.com/arp7/hadoop/commit/7b3212d2c41cc35cce81eadc68c029e0fc67a429
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit for the review and the detail feedbacks! I will address them in the next patch shortly.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit for the review and the detail feedbacks! I will address them in the next patch shortly.
          Hide
          xyao Xiaoyu Yao added a comment -

          Update patch v03 based on Arpit's feedback. Please review, thanks!

          DistributedFileSystem.java:2326: We can skip the call to dfs.getEZForPath if isHDFSEncryptionEnabled is false to avoid extra RPC call when TDE is not enabled.

          Good point. Fixed.

          FileSystem.java:2701: Can we define .Trash as a constant somewhere?

          Add FileSystem#TRASH_PREFIX for ".Trash"

          Trash.java:98: Avoid extra RPC for log statement. Can we cache the currentTrashDir some time earlier?

          Every path to be deleted may have different currentTrashDir. Move the INFO log to TrashPolicyDefault.java to avoid extra RPC for log.

          TrashPolicy.java:48: I don't think we should mark it as deprecated. While the TrashPolicyDefault no longer uses the home parameter other implementations may be passing a different value here in theory.

          TrashPolicy.java:57: Also we should have a default implementation of this routine else it will be a backward incompatible change (will break existing implementations of this public interface).
          TrashPolicy.java:83: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller.
          TrashPolicy.java:92: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller.

          Agree and fixed.

          TrashPolicy.java:108: We should leave the old method in place to keep the public interface backwards compatible. Perhaps to be conservative we should respect the 'home' parameter if one is passed in instead of using Filesystem#getTrashRoot?

          Agree and fixed.

          Show
          xyao Xiaoyu Yao added a comment - Update patch v03 based on Arpit 's feedback. Please review, thanks! DistributedFileSystem.java:2326: We can skip the call to dfs.getEZForPath if isHDFSEncryptionEnabled is false to avoid extra RPC call when TDE is not enabled. Good point. Fixed. FileSystem.java:2701: Can we define .Trash as a constant somewhere? Add FileSystem#TRASH_PREFIX for ".Trash" Trash.java:98: Avoid extra RPC for log statement. Can we cache the currentTrashDir some time earlier? Every path to be deleted may have different currentTrashDir. Move the INFO log to TrashPolicyDefault.java to avoid extra RPC for log. TrashPolicy.java:48: I don't think we should mark it as deprecated. While the TrashPolicyDefault no longer uses the home parameter other implementations may be passing a different value here in theory. TrashPolicy.java:57: Also we should have a default implementation of this routine else it will be a backward incompatible change (will break existing implementations of this public interface). TrashPolicy.java:83: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller. TrashPolicy.java:92: Need default implementation. It can just throw UnsupportedOperationException which should be handled by the caller. Agree and fixed. TrashPolicy.java:108: We should leave the old method in place to keep the public interface backwards compatible. Perhaps to be conservative we should respect the 'home' parameter if one is passed in instead of using Filesystem#getTrashRoot? Agree and fixed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 8m 53s trunk passed
          +1 compile 9m 18s trunk passed with JDK v1.8.0_66
          +1 compile 9m 34s trunk passed with JDK v1.7.0_85
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 2m 39s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 6m 12s trunk passed
          +1 javadoc 2m 18s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 13s trunk passed with JDK v1.7.0_85
          +1 mvninstall 3m 9s the patch passed
          +1 compile 8m 0s the patch passed with JDK v1.8.0_66
          -1 javac 18m 41s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new issues (was 752, now 753).
          +1 javac 8m 0s the patch passed
          +1 compile 8m 27s the patch passed with JDK v1.7.0_85
          -1 javac 27m 8s root-jdk1.7.0_85 with JDK v1.7.0_85 generated 5 new issues (was 745, now 746).
          +1 javac 8m 27s the patch passed
          -1 checkstyle 0m 57s Patch generated 11 new checkstyle issues in root (total was 220, now 226).
          +1 mvnsite 2m 27s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 1m 59s hadoop-common-project/hadoop-common introduced 2 new FindBugs issues.
          +1 javadoc 2m 16s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 16s the patch passed with JDK v1.7.0_85
          +1 unit 7m 15s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 52m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 7m 44s hadoop-common in the patch passed with JDK v1.7.0_85.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85.
          -1 unit 49m 29s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 20s Patch generated 58 ASF License warnings.
          200m 24s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Invocation of toString on trashRoots in org.apache.hadoop.fs.TrashPolicyDefault.createCheckpoint(Date) At TrashPolicyDefault.java:in org.apache.hadoop.fs.TrashPolicyDefault.createCheckpoint(Date) At TrashPolicyDefault.java:[line 180]
            Invocation of toString on trashRoots in org.apache.hadoop.fs.TrashPolicyDefault.deleteCheckpoint() At TrashPolicyDefault.java:in org.apache.hadoop.fs.TrashPolicyDefault.deleteCheckpoint() At TrashPolicyDefault.java:[line 190]
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReplication
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775403/HDFS-8831.03.patch
          JIRA Issue HDFS-8831
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0316920a1599 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6b9a5be
          findbugs v3.0.0
          javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt
          javac root-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_85.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13741/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: .
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13741/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 53s trunk passed +1 compile 9m 18s trunk passed with JDK v1.8.0_66 +1 compile 9m 34s trunk passed with JDK v1.7.0_85 +1 checkstyle 1m 4s trunk passed +1 mvnsite 2m 39s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 6m 12s trunk passed +1 javadoc 2m 18s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 13s trunk passed with JDK v1.7.0_85 +1 mvninstall 3m 9s the patch passed +1 compile 8m 0s the patch passed with JDK v1.8.0_66 -1 javac 18m 41s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new issues (was 752, now 753). +1 javac 8m 0s the patch passed +1 compile 8m 27s the patch passed with JDK v1.7.0_85 -1 javac 27m 8s root-jdk1.7.0_85 with JDK v1.7.0_85 generated 5 new issues (was 745, now 746). +1 javac 8m 27s the patch passed -1 checkstyle 0m 57s Patch generated 11 new checkstyle issues in root (total was 220, now 226). +1 mvnsite 2m 27s the patch passed +1 mvneclipse 0m 42s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 1m 59s hadoop-common-project/hadoop-common introduced 2 new FindBugs issues. +1 javadoc 2m 16s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 16s the patch passed with JDK v1.7.0_85 +1 unit 7m 15s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 51s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 52m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 7m 44s hadoop-common in the patch passed with JDK v1.7.0_85. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85. -1 unit 49m 29s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 20s Patch generated 58 ASF License warnings. 200m 24s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Invocation of toString on trashRoots in org.apache.hadoop.fs.TrashPolicyDefault.createCheckpoint(Date) At TrashPolicyDefault.java:in org.apache.hadoop.fs.TrashPolicyDefault.createCheckpoint(Date) At TrashPolicyDefault.java: [line 180]   Invocation of toString on trashRoots in org.apache.hadoop.fs.TrashPolicyDefault.deleteCheckpoint() At TrashPolicyDefault.java:in org.apache.hadoop.fs.TrashPolicyDefault.deleteCheckpoint() At TrashPolicyDefault.java: [line 190] JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReplication JDK v1.7.0_85 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775403/HDFS-8831.03.patch JIRA Issue HDFS-8831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0316920a1599 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6b9a5be findbugs v3.0.0 javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt javac root-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_85.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13741/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13741/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13741/console This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Fix the checkstyle, javac, findbugs and whitespace issues reported by Jenkins.

          Show
          xyao Xiaoyu Yao added a comment - Fix the checkstyle, javac, findbugs and whitespace issues reported by Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 9m 0s trunk passed
          +1 compile 10m 47s trunk passed with JDK v1.8.0_66
          +1 compile 11m 19s trunk passed with JDK v1.7.0_85
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 2m 59s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 6m 36s trunk passed
          +1 javadoc 2m 58s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85
          +1 mvninstall 3m 29s the patch passed
          +1 compile 10m 59s the patch passed with JDK v1.8.0_66
          -1 javac 23m 30s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new issues (was 751, now 752).
          +1 javac 10m 59s the patch passed
          +1 compile 10m 45s the patch passed with JDK v1.7.0_85
          -1 javac 34m 15s root-jdk1.7.0_85 with JDK v1.7.0_85 generated 6 new issues (was 745, now 746).
          +1 javac 10m 45s the patch passed
          -1 checkstyle 1m 12s Patch generated 3 new checkstyle issues in root (total was 220, now 218).
          +1 mvnsite 2m 59s the patch passed
          +1 mvneclipse 0m 48s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 7m 18s the patch passed
          +1 javadoc 2m 54s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 54s the patch passed with JDK v1.7.0_85
          -1 unit 8m 35s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 68m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 8m 51s hadoop-common in the patch passed with JDK v1.7.0_85.
          +1 unit 1m 11s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85.
          -1 unit 63m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 22s Patch generated 58 ASF License warnings.
          248m 7s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.ha.TestZKFailoverController
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.server.namenode.TestAuditLogs
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.namenode.TestCacheDirectives
          JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.tools.TestDFSHAAdmin
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
            hadoop.hdfs.TestAclsEndToEnd



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775510/HDFS-8831.04.patch
          JIRA Issue HDFS-8831
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dd31343447ca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3857fed
          findbugs v3.0.0
          javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt
          javac root-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_85.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13749/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: .
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13749/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 9m 0s trunk passed +1 compile 10m 47s trunk passed with JDK v1.8.0_66 +1 compile 11m 19s trunk passed with JDK v1.7.0_85 +1 checkstyle 1m 10s trunk passed +1 mvnsite 2m 59s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 6m 36s trunk passed +1 javadoc 2m 58s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 49s trunk passed with JDK v1.7.0_85 +1 mvninstall 3m 29s the patch passed +1 compile 10m 59s the patch passed with JDK v1.8.0_66 -1 javac 23m 30s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new issues (was 751, now 752). +1 javac 10m 59s the patch passed +1 compile 10m 45s the patch passed with JDK v1.7.0_85 -1 javac 34m 15s root-jdk1.7.0_85 with JDK v1.7.0_85 generated 6 new issues (was 745, now 746). +1 javac 10m 45s the patch passed -1 checkstyle 1m 12s Patch generated 3 new checkstyle issues in root (total was 220, now 218). +1 mvnsite 2m 59s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 7m 18s the patch passed +1 javadoc 2m 54s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 54s the patch passed with JDK v1.7.0_85 -1 unit 8m 35s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 68m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 8m 51s hadoop-common in the patch passed with JDK v1.7.0_85. +1 unit 1m 11s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85. -1 unit 63m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 22s Patch generated 58 ASF License warnings. 248m 7s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag   hadoop.ha.TestZKFailoverController   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.namenode.TestAuditLogs   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.namenode.TestCacheDirectives JDK v1.7.0_85 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.tools.TestDFSHAAdmin   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestAclsEndToEnd Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775510/HDFS-8831.04.patch JIRA Issue HDFS-8831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dd31343447ca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3857fed findbugs v3.0.0 javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt javac root-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_85.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13749/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13749/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13749/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Hi Xiaoyu, thanks for the updated patch. A few minor comments.

          1. FileSystem#getTrashRoots can return Collection<FileStatus> instead of FileStatus[]. Sorry I missed this on my last review.
          2. DistributedFileSystem.java:2329 - Use FileSystem.TRASH_PREFIX instead of ".Trash". Also lines 2356 and 2363.
          3. TrashPolicyDefault.java:58 - Use FileSystem.TRASH_PREFIX.
          4. Nitpick: FileSystem.java:108 - Need space before =.
          5. If you think it makes sense, we can add a Javadoc to TrashPolicy#getCurrentTrashDir(void) stating it may return an incorrect value if the EZ feature is enabled and the target file is in an EZ. And callers should prefer TrashPolicy#getCurrentTrashDir(Path path).

          +1 otherwise.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Hi Xiaoyu, thanks for the updated patch. A few minor comments. FileSystem#getTrashRoots can return Collection<FileStatus> instead of FileStatus[] . Sorry I missed this on my last review. DistributedFileSystem.java:2329 - Use FileSystem.TRASH_PREFIX instead of ".Trash". Also lines 2356 and 2363. TrashPolicyDefault.java:58 - Use FileSystem.TRASH_PREFIX. Nitpick: FileSystem.java:108 - Need space before = . If you think it makes sense, we can add a Javadoc to TrashPolicy#getCurrentTrashDir(void) stating it may return an incorrect value if the EZ feature is enabled and the target file is in an EZ. And callers should prefer TrashPolicy#getCurrentTrashDir(Path path) . +1 otherwise.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. Patch v05 addresses the latest review comments.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. Patch v05 addresses the latest review comments.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the v05 patch, pending Jenkins. Thanks Xiaoyu Yao.

          Trivial typo in TrashPolicy.java - reanme --> rename. Feel free to fix while committing.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the v05 patch, pending Jenkins. Thanks Xiaoyu Yao . Trivial typo in TrashPolicy.java - reanme --> rename . Feel free to fix while committing.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 14m 34s trunk passed
          +1 compile 8m 4s trunk passed with JDK v1.8.0_66
          +1 compile 9m 3s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 2s trunk passed
          +1 mvnsite 2m 33s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 5m 34s trunk passed
          +1 javadoc 2m 24s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 35s trunk passed with JDK v1.7.0_91
          -1 mvninstall 0m 33s hadoop-hdfs-client in the patch failed.
          -1 mvninstall 0m 48s hadoop-hdfs in the patch failed.
          +1 compile 7m 58s the patch passed with JDK v1.8.0_66
          -1 javac 18m 57s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new issues (was 752, now 753).
          +1 javac 7m 58s the patch passed
          +1 compile 8m 35s the patch passed with JDK v1.7.0_91
          -1 javac 27m 32s root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new issues (was 745, now 746).
          +1 javac 8m 35s the patch passed
          -1 checkstyle 1m 34s Patch generated 5 new checkstyle issues in root (total was 220, now 219).
          +1 mvnsite 2m 28s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 55s the patch passed
          +1 javadoc 2m 19s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 16s the patch passed with JDK v1.7.0_91
          +1 unit 7m 33s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 82m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 9m 1s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 1m 10s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 70m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 27s Patch generated 58 ASF License warnings.
          271m 33s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.server.datanode.TestBlockReplacement
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestBackupNode
            hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.TestEncryptionZonesWithKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775673/HDFS-8831.05.patch
          JIRA Issue HDFS-8831
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8173a87405a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 924a33d
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt
          javac root-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13761/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: .
          Max memory used 79MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13761/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 14m 34s trunk passed +1 compile 8m 4s trunk passed with JDK v1.8.0_66 +1 compile 9m 3s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 2s trunk passed +1 mvnsite 2m 33s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 5m 34s trunk passed +1 javadoc 2m 24s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 35s trunk passed with JDK v1.7.0_91 -1 mvninstall 0m 33s hadoop-hdfs-client in the patch failed. -1 mvninstall 0m 48s hadoop-hdfs in the patch failed. +1 compile 7m 58s the patch passed with JDK v1.8.0_66 -1 javac 18m 57s root-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new issues (was 752, now 753). +1 javac 7m 58s the patch passed +1 compile 8m 35s the patch passed with JDK v1.7.0_91 -1 javac 27m 32s root-jdk1.7.0_91 with JDK v1.7.0_91 generated 5 new issues (was 745, now 746). +1 javac 8m 35s the patch passed -1 checkstyle 1m 34s Patch generated 5 new checkstyle issues in root (total was 220, now 219). +1 mvnsite 2m 28s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 55s the patch passed +1 javadoc 2m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 16s the patch passed with JDK v1.7.0_91 +1 unit 7m 33s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 82m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 9m 1s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 1m 10s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 70m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 27s Patch generated 58 ASF License warnings. 271m 33s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.server.datanode.TestBlockReplacement JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestBackupNode   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.TestEncryptionZonesWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775673/HDFS-8831.05.patch JIRA Issue HDFS-8831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8173a87405a8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 924a33d findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt javac root-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_66.txt javac root-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13761/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13761/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 79MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13761/console This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          The recent Jenkins shows failure in these related tests. The first one can not repro on my local machine. The second test does not exist.

           org.apache.hadoop.hdfs.TestEncryptionZones.testEncryptionZoneWithTrash
           org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testEncryptionZoneWithTrash
          

          The mvninstall issue seems to infra issue unrelated to this change. I will commit the patch shortly.

          [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
          Failed while enforcing releasability the error(s) are [
          Dependency convergence error for org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT paths to dependency are:
          +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT
            +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
              +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT
          and
          +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT
            +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
              +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT
          and
          +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT
            +-org.apache.hadoop:hadoop-annotations:3.0.0-20151203.232546-7827]
          
          Show
          xyao Xiaoyu Yao added a comment - The recent Jenkins shows failure in these related tests. The first one can not repro on my local machine. The second test does not exist. org.apache.hadoop.hdfs.TestEncryptionZones.testEncryptionZoneWithTrash org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testEncryptionZoneWithTrash The mvninstall issue seems to infra issue unrelated to this change. I will commit the patch shortly. [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message: Failed while enforcing releasability the error(s) are [ Dependency convergence error for org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT paths to dependency are: +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT and +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT +-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT and +-org.apache.hadoop:hadoop-hdfs-client:3.0.0-SNAPSHOT +-org.apache.hadoop:hadoop-annotations:3.0.0-20151203.232546-7827]
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. I've commit the patch to the trunk.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. I've commit the patch to the trunk.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8922 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8922/)
          HDFS-8831. Trash Support for deletion in HDFS encryption zone. (xyao: rev cbc7b6bf97a80c39d4bbb3005e42dacae6726baf)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Trash.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8922 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8922/ ) HDFS-8831 . Trash Support for deletion in HDFS encryption zone. (xyao: rev cbc7b6bf97a80c39d4bbb3005e42dacae6726baf) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Trash.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #665 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/665/)
          HDFS-8831. Trash Support for deletion in HDFS encryption zone. (xyao: rev cbc7b6bf97a80c39d4bbb3005e42dacae6726baf)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicy.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Trash.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #665 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/665/ ) HDFS-8831 . Trash Support for deletion in HDFS encryption zone. (xyao: rev cbc7b6bf97a80c39d4bbb3005e42dacae6726baf) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicy.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Trash.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShell.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to branch-2.8.

          Show
          leftnoteasy Wangda Tan added a comment - Committed to branch-2.8.
          Hide
          zhz Zhe Zhang added a comment -

          Xiaoyu Yao Arpit Agarwal Wangda Tan Andrew Wang :

          Since the new getCurrentTrashDir method is throwing an IOException, it could break other applications using it. Should we mark this change as incompatible? Alternatively, we should look at how to safely handle the exception within HDFS.

          Show
          zhz Zhe Zhang added a comment - Xiaoyu Yao Arpit Agarwal Wangda Tan Andrew Wang : Since the new getCurrentTrashDir method is throwing an IOException, it could break other applications using it. Should we mark this change as incompatible? Alternatively, we should look at how to safely handle the exception within HDFS.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Hi Zhe Zhang, we should definitely fix it since we don't want an incompatible change in 2.8.0. DistributedFileSystem#getTrashRoot does throw so we should just fix it. We can remove the IncompatibleChange label after the fix. I will review your fix patch. Thanks for reporting this.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Hi Zhe Zhang , we should definitely fix it since we don't want an incompatible change in 2.8.0. DistributedFileSystem#getTrashRoot does throw so we should just fix it. We can remove the IncompatibleChange label after the fix. I will review your fix patch. Thanks for reporting this.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Arpit. I think we should also decide whether to revert this change and redo it (combined with HDFS-9799 change), or keep them as 2 separate commits. If keeping them separate, technically we still need to mark this as incompatible right?

          Show
          zhz Zhe Zhang added a comment - Thanks Arpit. I think we should also decide whether to revert this change and redo it (combined with HDFS-9799 change), or keep them as 2 separate commits. If keeping them separate, technically we still need to mark this as incompatible right?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          If keeping them separate, technically we still need to mark this as incompatible right?

          Marking it as incompatible will show up in release notes which will be wrong if we can fix it in time for the release. I added a "breaks" link instead.

          I think we should also decide whether to revert this change and redo it (combined with HDFS-9799 change)

          No need to revert if we can quickly fix HDFS-9799. If a fix looks impossible for 2.8.0 we can revert it. For now HDFS-9799 is rightly tagged as a blocker so there is no risk of missing it. Agreed?

          Show
          arpitagarwal Arpit Agarwal added a comment - If keeping them separate, technically we still need to mark this as incompatible right? Marking it as incompatible will show up in release notes which will be wrong if we can fix it in time for the release. I added a "breaks" link instead. I think we should also decide whether to revert this change and redo it (combined with HDFS-9799 change) No need to revert if we can quickly fix HDFS-9799 . If a fix looks impossible for 2.8.0 we can revert it. For now HDFS-9799 is rightly tagged as a blocker so there is no risk of missing it. Agreed?
          Hide
          zhz Zhe Zhang added a comment -

          Sounds good to me. I don't expect HDFS-9799 to take too long.

          Show
          zhz Zhe Zhang added a comment - Sounds good to me. I don't expect HDFS-9799 to take too long.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Made minor edits to the release note. Xiaoyu Yao, Zhe Zhang, do you think anything in the HDFS-8831 release note is invalidated by later fixes to EZ trash? (I did not follow the subsequent patches very closely).

          Show
          arpitagarwal Arpit Agarwal added a comment - Made minor edits to the release note. Xiaoyu Yao , Zhe Zhang , do you think anything in the HDFS-8831 release note is invalidated by later fixes to EZ trash? (I did not follow the subsequent patches very closely).

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development