Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently the lease manager records leases based on path instead of inode ids. Therefore, the lease manager needs to carefully keep track of the path of active leases during renames and deletes. This can be a non-trivial task.

      This jira proposes to simplify the logic by tracking leases using inodeids instead of paths.

      1. HDFS-6757.000.patch
        38 kB
        Haohui Mai
      2. HDFS-6757.001.patch
        39 kB
        Haohui Mai
      3. HDFS-6757.002.patch
        39 kB
        Haohui Mai
      4. HDFS-6757.003.patch
        39 kB
        Haohui Mai
      5. HDFS-6757.004.patch
        98 kB
        Haohui Mai
      6. HDFS-6757.005.patch
        96 kB
        Haohui Mai
      7. HDFS-6757.006.patch
        101 kB
        Haohui Mai
      8. HDFS-6757.007.patch
        101 kB
        Haohui Mai
      9. HDFS-6757.008.patch
        101 kB
        Haohui Mai
      10. HDFS-6757.009.patch
        52 kB
        Haohui Mai
      11. HDFS-6757.010.patch
        52 kB
        Haohui Mai
      12. HDFS-6757.011.patch
        95 kB
        Haohui Mai
      13. HDFS-6757.012.patch
        95 kB
        Haohui Mai
      14. HDFS-6757.013.patch
        95 kB
        Haohui Mai
      15. HDFS-6757.014.patch
        93 kB
        Haohui Mai
      16. HDFS-6757.015.patch
        93 kB
        Haohui Mai
      17. HDFS-6757.016.patch
        94 kB
        Haohui Mai
      18. HDFS-6757.017.patch
        94 kB
        Haohui Mai

        Activity

        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/)
        HDFS-6757. Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
          Add missing entry in CHANGES.txt for HDFS-6757. (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/ ) HDFS-6757 . Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java Add missing entry in CHANGES.txt for HDFS-6757 . (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/)
        HDFS-6757. Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          Add missing entry in CHANGES.txt for HDFS-6757. (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/ ) HDFS-6757 . Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java Add missing entry in CHANGES.txt for HDFS-6757 . (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/)
        HDFS-6757. Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Add missing entry in CHANGES.txt for HDFS-6757. (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/ ) HDFS-6757 . Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java Add missing entry in CHANGES.txt for HDFS-6757 . (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/)
        HDFS-6757. Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          Add missing entry in CHANGES.txt for HDFS-6757. (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/ ) HDFS-6757 . Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java Add missing entry in CHANGES.txt for HDFS-6757 . (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/)
        HDFS-6757. Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          Add missing entry in CHANGES.txt for HDFS-6757. (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/922/ ) HDFS-6757 . Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java Add missing entry in CHANGES.txt for HDFS-6757 . (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/)
        HDFS-6757. Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
          Add missing entry in CHANGES.txt for HDFS-6757. (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26)
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/ ) HDFS-6757 . Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java Add missing entry in CHANGES.txt for HDFS-6757 . (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
        +1 javac 7m 29s There were no new javac warning messages.
        +1 javadoc 9m 43s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 19s The applied patch generated 1364 new checkstyle issues (total was 3116, now 2707).
        -1 whitespace 2m 51s The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 40s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 15s Pre-build of native portion
        -1 hdfs tests 166m 6s Tests failed in hadoop-hdfs.
            212m 5s  



        Reason Tests
        Failed unit tests hadoop.hdfs.tools.TestHdfsConfigFields
          hadoop.tracing.TestTraceAdmin



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731640/HDFS-6757.017.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6471d18
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10892/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10892/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10892/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10892/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10892/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 43s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 19s The applied patch generated 1364 new checkstyle issues (total was 3116, now 2707). -1 whitespace 2m 51s The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 40s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 166m 6s Tests failed in hadoop-hdfs.     212m 5s   Reason Tests Failed unit tests hadoop.hdfs.tools.TestHdfsConfigFields   hadoop.tracing.TestTraceAdmin Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731640/HDFS-6757.017.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6471d18 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10892/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10892/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10892/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10892/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10892/console This message was automatically generated.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7783 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7783/)
        Add missing entry in CHANGES.txt for HDFS-6757. (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7783 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7783/ ) Add missing entry in CHANGES.txt for HDFS-6757 . (wheat9: rev 3becc3af8382caed2c3bf941f8fed6daf6e7bc26) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7782 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7782/)
        HDFS-6757. Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7782 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7782/ ) HDFS-6757 . Simplify lease manager with INodeID. Contributed by Haohui Mai. (wheat9: rev 00fe1ed3a4b3ee35fe24be257ec36445d2f44d63) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGetBlockLocations.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeFileUnderConstructionWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
        Hide
        Haohui Mai added a comment -

        I've committed the patch to trunk and branch-2. Thanks very much for all the reviews.

        Show
        Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks very much for all the reviews.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 40s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 14s The applied patch generated 20 new checkstyle issues (total was 1079, now 899).
        -1 whitespace 1m 21s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 38s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 7s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 14s Pre-build of native portion
        -1 hdfs tests 165m 23s Tests failed in hadoop-hdfs.
            209m 41s  



        Reason Tests
        Failed unit tests hadoop.tracing.TestTraceAdmin
          hadoop.hdfs.tools.TestHdfsConfigFields



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731640/HDFS-6757.017.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 315074b
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10886/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10886/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10886/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10886/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10886/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 20 new checkstyle issues (total was 1079, now 899). -1 whitespace 1m 21s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 7s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 165m 23s Tests failed in hadoop-hdfs.     209m 41s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin   hadoop.hdfs.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731640/HDFS-6757.017.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 315074b checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10886/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10886/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10886/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10886/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10886/console This message was automatically generated.
        Hide
        Jing Zhao added a comment -

        The 016 patch looks pretty good to me. But the IOException when applying editlog is still there. Other than this +1.

        Show
        Jing Zhao added a comment - The 016 patch looks pretty good to me. But the IOException when applying editlog is still there. Other than this +1.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 52s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
        +1 javac 7m 29s There were no new javac warning messages.
        +1 javadoc 9m 41s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 49s There were no new checkstyle issues.
        -1 whitespace 0m 26s The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 13s Pre-build of native portion
        -1 hdfs tests 165m 15s Tests failed in hadoop-hdfs.
            207m 33s  



        Reason Tests
        Failed unit tests hadoop.tracing.TestTraceAdmin
          hadoop.hdfs.TestRollingUpgrade



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731505/HDFS-6757.016.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a0fae8d
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10878/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10878/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10878/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10878/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 49s There were no new checkstyle issues. -1 whitespace 0m 26s The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 13s Pre-build of native portion -1 hdfs tests 165m 15s Tests failed in hadoop-hdfs.     207m 33s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin   hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731505/HDFS-6757.016.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a0fae8d whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10878/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10878/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10878/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10878/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 24s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
        -1 javac 1m 53s The patch appears to cause the build to fail.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731289/HDFS-6757.014.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 4d9f9e5
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10852/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. -1 javac 1m 53s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731289/HDFS-6757.014.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4d9f9e5 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10852/console This message was automatically generated.
        Hide
        Haohui Mai added a comment - - edited

        Thanks Jing for the reviews. The v14 patch addresses all comments except 7 and 11.

        7. One question is, after we record INode id in lease manager, do we still want to record full paths of UC files in FSImage?

        It's a good proposal. I'll address it in a separate jira.

        11. Maybe we can add extra tests for the delete and rename operations when UC files are involved. Specifically, a test on the internal lease recovery for renamed UC files may be useful.

        It looks like that TestLease has covered this case.

        Show
        Haohui Mai added a comment - - edited Thanks Jing for the reviews. The v14 patch addresses all comments except 7 and 11. 7. One question is, after we record INode id in lease manager, do we still want to record full paths of UC files in FSImage? It's a good proposal. I'll address it in a separate jira. 11. Maybe we can add extra tests for the delete and rename operations when UC files are involved. Specifically, a test on the internal lease recovery for renamed UC files may be useful. It looks like that TestLease has covered this case.
        Hide
        Jing Zhao added a comment -

        Thanks for continuing the work here, Haohui. Some comments on the latest 013 patch:

        1. Any reason to use o1.lastUpdate but use o2.getLastUpdate() in the
          follwoing code?
          +  private final PriorityQueue<Lease> sortedLeases = new PriorityQueue<Lease>(512,
          +      new Comparator<Lease>() {
          +        @Override
          +        public int compare(Lease o1, Lease o2) {
          +          return Long.signum(o1.lastUpdate - o2.getLastUpdate());
          +        }
          +  });
          
        2. Let's keep this Precondition check in getNumUnderConstructionBlocks considering this patch is a big change.
          -          Preconditions.checkState(cons.isUnderConstruction());
          
        3. Similarly let's put this check into serializeFilesUCSection or print a warning msg if the inode is not UC.
          Preconditions.checkState(node.isUnderConstruction());
          
        4. What is the difference between removeLeases and removeLeaseById?
        5. We'd better add a check to make sure the iip and p are both valid,
          i.e., starting from root, to prevent a wrong path written
          into the CloseOp editlog.
          INodesInPath iip = INodesInPath.fromINode(fsd.getInode(id));
          p = iip.getPath();
          boolean completed = fsnamesystem.internalReleaseLease(
              leaseToCheck, p, iip,
              HdfsServerConstants.NAMENODE_LEASE_HOLDER);
          
        6. Any reason to adding this IOException for loading editlog?
                if (file.isUnderConstruction()) {
                  if (fsNamesys.leaseManager.getLease(file) == null) {
                    throw new IOException("UC file " + path + " does not have a " +
                            "corresponding lease");
                  }
          
        7. One question is, after we record INode id in lease manager, do we still want
          to record full paths of UC files in FSImage?
        8. In INodeFile#destroyAndCollectBlocks, which may also be called when
          deleting a snapshot, removedUCFiles may be null? If yes, we need a unit test to catch the case.
              FileUnderConstructionFeature uc = getFileUnderConstructionFeature();
              if (uc != null) {
                removedUCFiles.add(getId());
              }
          
        9. When INodeFile#cleanSubtree is called because the file or its ancestor is deleted,
          even if the file is in snapshot, we should check its UC state and update removedUCFiles accordingly.
        10. The changes in FileDiff is not needed.
        11. Maybe we can add extra tests for the delete and rename operations when UC files are involved. Specifically, a test on the internal lease recovery for renamed UC files may be useful.
        12. Nit: the following code needs reformat:
          +  private final HashMap<Long, Lease> leasesById = new
          +          HashMap<>();
          

          and

          LOG.debug("Lease recovery for inode " + id + " is complete. " +
                  "File" +
                  " closed.");
          

          and

          -   *
          -   * @param bsps
          +   *  @param bsps
          
        Show
        Jing Zhao added a comment - Thanks for continuing the work here, Haohui. Some comments on the latest 013 patch: Any reason to use o1.lastUpdate but use o2.getLastUpdate() in the follwoing code? + private final PriorityQueue<Lease> sortedLeases = new PriorityQueue<Lease>(512, + new Comparator<Lease>() { + @Override + public int compare(Lease o1, Lease o2) { + return Long .signum(o1.lastUpdate - o2.getLastUpdate()); + } + }); Let's keep this Precondition check in getNumUnderConstructionBlocks considering this patch is a big change. - Preconditions.checkState(cons.isUnderConstruction()); Similarly let's put this check into serializeFilesUCSection or print a warning msg if the inode is not UC. Preconditions.checkState(node.isUnderConstruction()); What is the difference between removeLeases and removeLeaseById ? We'd better add a check to make sure the iip and p are both valid, i.e., starting from root, to prevent a wrong path written into the CloseOp editlog. INodesInPath iip = INodesInPath.fromINode(fsd.getInode(id)); p = iip.getPath(); boolean completed = fsnamesystem.internalReleaseLease( leaseToCheck, p, iip, HdfsServerConstants.NAMENODE_LEASE_HOLDER); Any reason to adding this IOException for loading editlog? if (file.isUnderConstruction()) { if (fsNamesys.leaseManager.getLease(file) == null ) { throw new IOException( "UC file " + path + " does not have a " + "corresponding lease" ); } One question is, after we record INode id in lease manager, do we still want to record full paths of UC files in FSImage? In INodeFile#destroyAndCollectBlocks , which may also be called when deleting a snapshot, removedUCFiles may be null? If yes, we need a unit test to catch the case. FileUnderConstructionFeature uc = getFileUnderConstructionFeature(); if (uc != null ) { removedUCFiles.add(getId()); } When INodeFile#cleanSubtree is called because the file or its ancestor is deleted, even if the file is in snapshot, we should check its UC state and update removedUCFiles accordingly. The changes in FileDiff is not needed. Maybe we can add extra tests for the delete and rename operations when UC files are involved. Specifically, a test on the internal lease recovery for renamed UC files may be useful. Nit: the following code needs reformat: + private final HashMap< Long , Lease> leasesById = new + HashMap<>(); and LOG.debug( "Lease recovery for inode " + id + " is complete. " + "File" + " closed." ); and - * - * @param bsps + * @param bsps
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 40s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 16s The applied patch generated 22 new checkstyle issues (total was 904, now 907).
        -1 whitespace 0m 19s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 native 3m 15s Pre-build of native portion
        +1 hdfs tests 165m 24s Tests passed in hadoop-hdfs.
            208m 47s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729861/HDFS-6757.013.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6f541ed
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10513/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10513/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10513/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10513/testReport/
        Java 1.7.0_55
        uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10513/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 16s The applied patch generated 22 new checkstyle issues (total was 904, now 907). -1 whitespace 0m 19s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 5s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 15s Pre-build of native portion +1 hdfs tests 165m 24s Tests passed in hadoop-hdfs.     208m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729861/HDFS-6757.013.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6f541ed checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10513/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10513/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10513/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10513/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10513/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 32s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 34s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 7m 46s The applied patch generated 9 additional checkstyle issues.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 3m 8s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.
        +1 native 3m 12s Pre-build of native portion
        -1 hdfs tests 223m 58s Tests failed in hadoop-hdfs.
            272m 12s  



        Reason Tests
        FindBugs module:hadoop-hdfs
          Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java:[lines 177-201]
          Dead store to src in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.prepareFileForTruncate(INodesInPath, String, String, long, Block) At FSNamesystem.java:org.apache.hadoop.hdfs.server.namenode.FSNamesystem.prepareFileForTruncate(INodesInPath, String, String, long, Block) At FSNamesystem.java:[line 2086]
        Failed unit tests hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.TestDFSOutputStream
          hadoop.hdfs.server.namenode.TestDeleteRace
          hadoop.hdfs.TestClose
          hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
          hadoop.hdfs.server.datanode.TestBlockRecovery
          hadoop.hdfs.TestFileCreationDelete
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.cli.TestHDFSCLI
          hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.TestMultiThreadedHflush
          hadoop.hdfs.TestQuota
          hadoop.hdfs.TestFileLengthOnClusterRestart
        Timed out tests org.apache.hadoop.hdfs.TestDataTransferProtocol
          org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729596/HDFS-6757.012.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 7e8639f
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10486/artifact/patchprocess/checkstyle-result-diff.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10486/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10486/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10486/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10486/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 7m 46s The applied patch generated 9 additional checkstyle issues. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 3m 8s The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 native 3m 12s Pre-build of native portion -1 hdfs tests 223m 58s Tests failed in hadoop-hdfs.     272m 12s   Reason Tests FindBugs module:hadoop-hdfs   Class org.apache.hadoop.hdfs.DataStreamer$LastException is not derived from an Exception, even though it is named as such At DataStreamer.java:from an Exception, even though it is named as such At DataStreamer.java: [lines 177-201]   Dead store to src in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.prepareFileForTruncate(INodesInPath, String, String, long, Block) At FSNamesystem.java:org.apache.hadoop.hdfs.server.namenode.FSNamesystem.prepareFileForTruncate(INodesInPath, String, String, long, Block) At FSNamesystem.java: [line 2086] Failed unit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestDFSOutputStream   hadoop.hdfs.server.namenode.TestDeleteRace   hadoop.hdfs.TestClose   hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation   hadoop.hdfs.server.datanode.TestBlockRecovery   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.cli.TestHDFSCLI   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestMultiThreadedHflush   hadoop.hdfs.TestQuota   hadoop.hdfs.TestFileLengthOnClusterRestart Timed out tests org.apache.hadoop.hdfs.TestDataTransferProtocol   org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery   org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache   org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729596/HDFS-6757.012.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7e8639f checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10486/artifact/patchprocess/checkstyle-result-diff.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/10486/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10486/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10486/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10486/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 57s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        -1 javac 1m 34s The patch appears to cause the build to fail.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729565/HDFS-6757.011.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e2e8f77
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10483/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 1m 34s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729565/HDFS-6757.011.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e2e8f77 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10483/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 55s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 8 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        -1 javac 1m 34s The patch appears to cause the build to fail.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729365/HDFS-6757.010.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / aa22450
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10465/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 55s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 8 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 1m 34s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729365/HDFS-6757.010.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / aa22450 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10465/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 8 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        -1 javac 1m 32s The patch appears to cause the build to fail.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729353/HDFS-6757.009.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 4c1af15
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10462/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 8 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 1m 32s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729353/HDFS-6757.009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4c1af15 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10462/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12679938/HDFS-6757.008.patch
        against trunk revision 2cc868d.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestFileAppend2

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8710//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8710//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679938/HDFS-6757.008.patch against trunk revision 2cc868d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestFileAppend2 +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8710//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8710//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12679938/HDFS-6757.008.patch
        against trunk revision 1670578.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 13 warning messages.
        See https://builds.apache.org/job/PreCommit-HDFS-Build/8680//artifact/patchprocess/diffJavadocWarnings.txt for details.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestSetTimes
        org.apache.hadoop.hdfs.TestDFSClientRetries

        The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8680//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8680//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679938/HDFS-6757.008.patch against trunk revision 1670578. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 13 warning messages. See https://builds.apache.org/job/PreCommit-HDFS-Build/8680//artifact/patchprocess/diffJavadocWarnings.txt for details. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.TestDFSClientRetries The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8680//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8680//console This message is automatically generated.
        Hide
        Haohui Mai added a comment -

        Rebased onto the latest trunk.

        Show
        Haohui Mai added a comment - Rebased onto the latest trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12667441/HDFS-6757.007.patch
        against trunk revision 0de563a.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7979//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7979//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667441/HDFS-6757.007.patch against trunk revision 0de563a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7979//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7979//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12667441/HDFS-6757.007.patch
        against trunk revision 90c8ece.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestRollingUpgradeRollback

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7969//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7969//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667441/HDFS-6757.007.patch against trunk revision 90c8ece. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestRollingUpgradeRollback +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7969//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7969//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12667441/HDFS-6757.007.patch
        against trunk revision 90c8ece.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7968//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12667441/HDFS-6757.007.patch against trunk revision 90c8ece. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7968//console This message is automatically generated.
        Hide
        Haohui Mai added a comment -

        The v7 patch is rebased onto the trunk.

        Show
        Haohui Mai added a comment - The v7 patch is rebased onto the trunk.
        Hide
        Daryn Sharp added a comment -

        The patch doesn't appear to apply anymore.

        Show
        Daryn Sharp added a comment - The patch doesn't appear to apply anymore.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666584/HDFS-6757.006.patch
        against trunk revision b44b2ee.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7902//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7902//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666584/HDFS-6757.006.patch against trunk revision b44b2ee. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7902//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7902//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12666572/HDFS-6757.005.patch
        against trunk revision b44b2ee.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7900//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666572/HDFS-6757.005.patch against trunk revision b44b2ee. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7900//console This message is automatically generated.
        Hide
        Haohui Mai added a comment -

        The v5 patch rebased on the latest trunk and addressed Colin's comments.

        I also verified that CloseOp stored the full path instead of the inode id.

        Show
        Haohui Mai added a comment - The v5 patch rebased on the latest trunk and addressed Colin's comments. I also verified that CloseOp stored the full path instead of the inode id.
        Hide
        Colin Patrick McCabe added a comment -

        So far I plan to change the code so that the NN bails out when it fails to replay the OP_CLOSE operations?

        Sounds good.

        Colin Patrick McCabe, do you have anything specific that you want to address?

        I think we should continue to store the path in the op, as Daryn suggested. This will make inotify more useful, since we'll have path information, while still allowing us to get the benefits of increased efficiency by lookup-by-inode ID.

        Show
        Colin Patrick McCabe added a comment - So far I plan to change the code so that the NN bails out when it fails to replay the OP_CLOSE operations? Sounds good. Colin Patrick McCabe, do you have anything specific that you want to address? I think we should continue to store the path in the op, as Daryn suggested. This will make inotify more useful, since we'll have path information, while still allowing us to get the benefits of increased efficiency by lookup-by-inode ID.
        Hide
        Haohui Mai added a comment -

        Thanks Colin Patrick McCabe and Daryn Sharp for the comments.

        Just to make sure that I don't accidentally miss anything, can you guys list the issues I need to address in the next patch? So far I plan to change the code so that the NN bails out when it fails to replay the OP_CLOSE operations? Colin Patrick McCabe, do you have anything specific that you want to address?

        Show
        Haohui Mai added a comment - Thanks Colin Patrick McCabe and Daryn Sharp for the comments. Just to make sure that I don't accidentally miss anything, can you guys list the issues I need to address in the next patch? So far I plan to change the code so that the NN bails out when it fails to replay the OP_CLOSE operations? Colin Patrick McCabe , do you have anything specific that you want to address?
        Hide
        Daryn Sharp added a comment -

        Should successfully replaying OP_CLOSE require that a corresponding lease be found? It hasn't in the past. [...] I guess perhaps we could say that OP_CLOSE is reserved for normal, clean file close operations

        As best I can tell we do currently require a lease to close and an edit op is generated only for clean closes. completeFileInternal throws if there is no lease but special logic for rpc retries will discard the lease exception and return success if the client is closing the last block with its current genstamp. However, no edit is generated so it's not a consideration here.

        A close op may also be generated by commit block sync, but it looks like the NN should have already reassigned the lease to itself during block recovery. I believe the design is a UC file must always have a lease of a client or the NN.

        The general concern is whether it's prudent to start masking possible bugs during edit replay. It makes me very uncomfortable. I'd rather the NN fail while processing illegal edit sequences because edit bugs can lead to data loss. Otherwise we are implicitly willing to let the standby silently become inconsistent with the active.

        Show
        Daryn Sharp added a comment - Should successfully replaying OP_CLOSE require that a corresponding lease be found? It hasn't in the past. [...] I guess perhaps we could say that OP_CLOSE is reserved for normal, clean file close operations As best I can tell we do currently require a lease to close and an edit op is generated only for clean closes. completeFileInternal throws if there is no lease but special logic for rpc retries will discard the lease exception and return success if the client is closing the last block with its current genstamp. However, no edit is generated so it's not a consideration here. A close op may also be generated by commit block sync, but it looks like the NN should have already reassigned the lease to itself during block recovery. I believe the design is a UC file must always have a lease of a client or the NN. The general concern is whether it's prudent to start masking possible bugs during edit replay. It makes me very uncomfortable. I'd rather the NN fail while processing illegal edit sequences because edit bugs can lead to data loss. Otherwise we are implicitly willing to let the standby silently become inconsistent with the active.
        Hide
        Colin Patrick McCabe added a comment -

        Daryn Sharp: you make an interesting point about OP_CLOSE. Should successfully replaying OP_CLOSE require that a corresponding lease be found? It hasn't in the past. This seems related to your other point about using references rather than inode IDs in the lease manager. If OP_CLOSE absolutely, positively has to remove those leases, then I'd be more OK with using references there. In other words, the LeaseManager has always been somewhat arms-length from the namesystem, but you are proposing to integrate them more closely.

        I thought the close fails if the lease is expired? If yes, my question is whether replaying edit logs should tolerate a close with no lease if the live NN does not allow it.

        Not exactly. See HDFS-3031. FSNamesystem#completeFile is actually an idempotent RPC that can succeed as many times as you like... under the right conditions. No lease required.

        In general nobody is very happy with how we handle errors closing files (see HDFS-4504 for details). It's still a bit unclear what the solution is. I guess perhaps we could say that OP_CLOSE is reserved for normal, clean file close operations, and lease expirations or reassignments get different ops. I have to think about whether we would ever want to do a "dirty" close via OP_CLOSE... I can't think of any scenario off the top of my head, but maybe you guys can?

        Show
        Colin Patrick McCabe added a comment - Daryn Sharp : you make an interesting point about OP_CLOSE. Should successfully replaying OP_CLOSE require that a corresponding lease be found? It hasn't in the past. This seems related to your other point about using references rather than inode IDs in the lease manager. If OP_CLOSE absolutely, positively has to remove those leases, then I'd be more OK with using references there. In other words, the LeaseManager has always been somewhat arms-length from the namesystem, but you are proposing to integrate them more closely. I thought the close fails if the lease is expired? If yes, my question is whether replaying edit logs should tolerate a close with no lease if the live NN does not allow it. Not exactly. See HDFS-3031 . FSNamesystem#completeFile is actually an idempotent RPC that can succeed as many times as you like... under the right conditions. No lease required. In general nobody is very happy with how we handle errors closing files (see HDFS-4504 for details). It's still a bit unclear what the solution is. I guess perhaps we could say that OP_CLOSE is reserved for normal, clean file close operations, and lease expirations or reassignments get different ops. I have to think about whether we would ever want to do a "dirty" close via OP_CLOSE... I can't think of any scenario off the top of my head, but maybe you guys can?
        Hide
        Colin Patrick McCabe added a comment -

        Perhaps I'm understanding. What is the use case for allowing a client to continue writing to a deleted file and have the results reflected in a snapshot? The lease is effectively for the mutable/non-snapshot path. If it's deleted, why wouldn't we want to revoke the lease? It's already "bad" enough that snapshotting a UC file does not record the visible length so the file in the snapshot continue to mutate, but continuing to modify a deleted file in a snapshot?

        Sorry, I guess I phrased this poorly. I'm not talking about writing to a snapshotted file, or anything like that. I'm just commenting on the implementation detail that led to the need for the removedUCFiles list. Basically, the issue is that an inode that is under construction may be part of a snapshot. Just because we remove the snapshot doesn't mean we should revoke the lease on the under-construction file. So we can't just do "if inode.isUnderconstruction { revoke_lease }" inside the deletion functions. We only want to revoke the lease when closing the file or removing it from the current reality.

        My mild concern/question is let's say this change is in 2.6. We deploy it, run a week, find a horrible problem and back up to 2.5 because it has the same layout.

        2.5 can't replay edits from 2.6. There's no forward-compatibility with edit logs, only backwards compatibility. Basically the way things work in the rolling upgrade universe is that some changes can be downgraded cleanly, and others require "rollback" (i.e. revert the data back to the previous version). HDFS-6757 is clearly a change that could be done via downgrade rather than rollback, since it's not adding any new data that 2.5 can't understand.

        Show
        Colin Patrick McCabe added a comment - Perhaps I'm understanding. What is the use case for allowing a client to continue writing to a deleted file and have the results reflected in a snapshot? The lease is effectively for the mutable/non-snapshot path. If it's deleted, why wouldn't we want to revoke the lease? It's already "bad" enough that snapshotting a UC file does not record the visible length so the file in the snapshot continue to mutate, but continuing to modify a deleted file in a snapshot? Sorry, I guess I phrased this poorly. I'm not talking about writing to a snapshotted file, or anything like that. I'm just commenting on the implementation detail that led to the need for the removedUCFiles list. Basically, the issue is that an inode that is under construction may be part of a snapshot. Just because we remove the snapshot doesn't mean we should revoke the lease on the under-construction file. So we can't just do "if inode.isUnderconstruction { revoke_lease }" inside the deletion functions. We only want to revoke the lease when closing the file or removing it from the current reality. My mild concern/question is let's say this change is in 2.6. We deploy it, run a week, find a horrible problem and back up to 2.5 because it has the same layout. 2.5 can't replay edits from 2.6. There's no forward-compatibility with edit logs, only backwards compatibility. Basically the way things work in the rolling upgrade universe is that some changes can be downgraded cleanly, and others require "rollback" (i.e. revert the data back to the previous version). HDFS-6757 is clearly a change that could be done via downgrade rather than rollback, since it's not adding any new data that 2.5 can't understand.
        Hide
        Daryn Sharp added a comment -

        Trying to catch up...

        I think it's fine to use INode in the lease map. However, in the meanwhile, I'm not sure if this will increase the possibility of memory leak due to failing to update the lease map (maybe because of buggy code).

        I agree-- we might get some really nasty bugs by having a bunch of references to inodes that somehow are "really" deleted here. It would only require a minor mistake in the code. The memory savings doesn't seem worth it (and, of course, we're already reducing memory consumption by switching away from strings)

        Yes, but minor bugs anywhere can have devastating results. The block manager doesn't make attempts to add indirection just in case there's a future bug. Even if there was a leak, lease expiration will eventually clean it up?

        My thinking is that even though the lease manager is buggy, NN should continue to function when replay edit logs / saving namespaces instead of crashing. Though not ideal, but it allows NN to tolerate the bugs and to treat them as lease recoveries. [...] I'm updating my patch to print out warnings to help developers to catch these bugs.

        I can agree to saving the namespace being tolerant. We would have been in a world of hurt after the recent edit log corruption bugs if we couldn't save the namespace on the active.

        I'm not comfortable with being tolerant of buggy edits. If the NN cannot re-do what it thinks it did, that's pretty serious, and should continue to be a stop the world event. The only reason we noticed the recent edit log corruptions is because the standby died. Sadly, nobody is going to look for or notice log messages until a major service interruption has occurred.

        The idea that the NN should be tolerant of its own bugs will inevitably lead to namespace and/or data corruption...

        Jing Zhao's made a good point about how we don't want to remove leases for open files just because we deleted an under-construction inode inside a snapshot.

        Perhaps I'm understanding. What is the use case for allowing a client to continue writing to a deleted file and have the results reflected in a snapshot? The lease is effectively for the mutable/non-snapshot path. If it's deleted, why wouldn't we want to revoke the lease? It's already "bad" enough that snapshotting a UC file does not record the visible length so the file in the snapshot continue to mutate, but continuing to modify a deleted file in a snapshot?

        Imagine that a client gets SIGSTOP, and then its hard lease on its open-for-write file expires and is removed. The client can be resumed later, come back and then try to close. So not being able to find a lease during a close is certainly worthy of a nasty log message, but I don't think we can call it a bug.

        I thought the close fails if the lease is expired? If yes, my question is whether replaying edit logs should tolerate a close with no lease if the live NN does not allow it.

        You're right that the new lease manager won't (and doesn't need to) update the lease during the renames, but I'm not sure that I fully understand your comments.

        My mild concern/question is let's say this change is in 2.6. We deploy it, run a week, find a horrible problem and back up to 2.5 because it has the same layout. If 2.5 replays the edits, will it be in a sane state due to the missing lease reassigns? I wish I had time to study the code and think about it, but I don't, so I'm posing the question.

        Show
        Daryn Sharp added a comment - Trying to catch up... I think it's fine to use INode in the lease map. However, in the meanwhile, I'm not sure if this will increase the possibility of memory leak due to failing to update the lease map (maybe because of buggy code). I agree-- we might get some really nasty bugs by having a bunch of references to inodes that somehow are "really" deleted here. It would only require a minor mistake in the code. The memory savings doesn't seem worth it (and, of course, we're already reducing memory consumption by switching away from strings) Yes, but minor bugs anywhere can have devastating results. The block manager doesn't make attempts to add indirection just in case there's a future bug. Even if there was a leak, lease expiration will eventually clean it up? My thinking is that even though the lease manager is buggy, NN should continue to function when replay edit logs / saving namespaces instead of crashing. Though not ideal, but it allows NN to tolerate the bugs and to treat them as lease recoveries. [...] I'm updating my patch to print out warnings to help developers to catch these bugs. I can agree to saving the namespace being tolerant. We would have been in a world of hurt after the recent edit log corruption bugs if we couldn't save the namespace on the active. I'm not comfortable with being tolerant of buggy edits. If the NN cannot re-do what it thinks it did, that's pretty serious, and should continue to be a stop the world event. The only reason we noticed the recent edit log corruptions is because the standby died. Sadly, nobody is going to look for or notice log messages until a major service interruption has occurred. The idea that the NN should be tolerant of its own bugs will inevitably lead to namespace and/or data corruption... Jing Zhao's made a good point about how we don't want to remove leases for open files just because we deleted an under-construction inode inside a snapshot. Perhaps I'm understanding. What is the use case for allowing a client to continue writing to a deleted file and have the results reflected in a snapshot? The lease is effectively for the mutable/non-snapshot path. If it's deleted, why wouldn't we want to revoke the lease? It's already "bad" enough that snapshotting a UC file does not record the visible length so the file in the snapshot continue to mutate, but continuing to modify a deleted file in a snapshot? Imagine that a client gets SIGSTOP, and then its hard lease on its open-for-write file expires and is removed. The client can be resumed later, come back and then try to close. So not being able to find a lease during a close is certainly worthy of a nasty log message, but I don't think we can call it a bug. I thought the close fails if the lease is expired? If yes, my question is whether replaying edit logs should tolerate a close with no lease if the live NN does not allow it. You're right that the new lease manager won't (and doesn't need to) update the lease during the renames, but I'm not sure that I fully understand your comments. My mild concern/question is let's say this change is in 2.6. We deploy it, run a week, find a horrible problem and back up to 2.5 because it has the same layout. If 2.5 replays the edits, will it be in a sane state due to the missing lease reassigns? I wish I had time to study the code and think about it, but I don't, so I'm posing the question.
        Hide
        Colin Patrick McCabe added a comment -

        Daryn Sharp: yeah, we could provide both an inode ID and a path in the close op. Maybe that's the best option here...

        Show
        Colin Patrick McCabe added a comment - Daryn Sharp : yeah, we could provide both an inode ID and a path in the close op. Maybe that's the best option here...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12658724/HDFS-6757.004.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7517//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7517//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658724/HDFS-6757.004.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7517//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7517//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12658724/HDFS-6757.004.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7511//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7511//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658724/HDFS-6757.004.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7511//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7511//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        Ah. I see what you're saying. We don't remove the inode at all if it's in an older snapshot, so we don't get a chance to remove the lease manager entry just by looking at removedINodes. Thanks for the explanation.

        Show
        Colin Patrick McCabe added a comment - Ah. I see what you're saying. We don't remove the inode at all if it's in an older snapshot, so we don't get a chance to remove the lease manager entry just by looking at removedINodes. Thanks for the explanation.
        Hide
        Haohui Mai added a comment -

        But we should be able to tell whether or not we're not removing the "current" version of the UC inode somehow, right?

        if (priorSnapshotId == NO_SNAPSHOT_ID) {
                // this only happens when deleting the current file and the file is not
                // in any snapshot
                computeQuotaUsage(counts, false);
                destroyAndCollectBlocks(collectedBlocks, removedINodes, removedUCFiles);
              } else {
                // when deleting the current file and the file is in snapshot, we should
                // clean the 0-sized block if the file is UC
                FileUnderConstructionFeature uc = getFileUnderConstructionFeature();
                if (uc != null) {
                  uc.cleanZeroSizeBlock(this, collectedBlocks);
                  if (removedUCFiles != null) {
                    removedUCFiles.add(this);
                  }
                }
              }
        

        where an inode is only put into removedINode if it does not reside in any snapshots. Therefore unfortunately it seems that the implementation needs removedUCFiles to count the UCfiles right.

        Show
        Haohui Mai added a comment - But we should be able to tell whether or not we're not removing the "current" version of the UC inode somehow, right? if (priorSnapshotId == NO_SNAPSHOT_ID) { // this only happens when deleting the current file and the file is not // in any snapshot computeQuotaUsage(counts, false ); destroyAndCollectBlocks(collectedBlocks, removedINodes, removedUCFiles); } else { // when deleting the current file and the file is in snapshot, we should // clean the 0-sized block if the file is UC FileUnderConstructionFeature uc = getFileUnderConstructionFeature(); if (uc != null ) { uc.cleanZeroSizeBlock( this , collectedBlocks); if (removedUCFiles != null ) { removedUCFiles.add( this ); } } } where an inode is only put into removedINode if it does not reside in any snapshots. Therefore unfortunately it seems that the implementation needs removedUCFiles to count the UCfiles right.
        Hide
        Colin Patrick McCabe added a comment -

        Jing Zhao wrote: With INode features we now no longer have INode replacement (exception for INodeReference which is out of picture in this jira), thus I think it's fine to use INode in the lease map. However, in the meanwhile, I'm not sure if this will increase the possibility of memory leak due to failing to update the lease map (maybe because of buggy code).

        I agree-- we might get some really nasty bugs by having a bunch of references to inodes that somehow are "really" deleted here. It would only require a minor mistake in the code. The memory savings doesn't seem worth it (and, of course, we're already reducing memory consumption by switching away from strings)

        Daryn Sharp wrote: However, isn't a bug if there's no lease during a close op? Perhaps removeLeases skips missing leases and calls a removeLease that requires a lease, and the close edit op calls removeLease?

        Imagine that a client gets SIGSTOP, and then its hard lease on its open-for-write file expires and is removed. The client can be resumed later, come back and then try to close. So not being able to find a lease during a close is certainly worthy of a nasty log message, but I don't think we can call it a bug.

        Haohui Mai: I apologize if it is a dumb question, but do we need to pass removedUCFiles around everywhere? Can't we just look at the inodes in the lists we're passing and see if they're under construction? Jing Zhao's made a good point about how we don't want to remove leases for open files just because we deleted an under-construction inode inside a snapshot. But we should be able to tell whether or not we're not removing the "current" version of the UC inode somehow, right?

        Show
        Colin Patrick McCabe added a comment - Jing Zhao wrote: With INode features we now no longer have INode replacement (exception for INodeReference which is out of picture in this jira), thus I think it's fine to use INode in the lease map. However, in the meanwhile, I'm not sure if this will increase the possibility of memory leak due to failing to update the lease map (maybe because of buggy code). I agree-- we might get some really nasty bugs by having a bunch of references to inodes that somehow are "really" deleted here. It would only require a minor mistake in the code. The memory savings doesn't seem worth it (and, of course, we're already reducing memory consumption by switching away from strings) Daryn Sharp wrote: However, isn't a bug if there's no lease during a close op? Perhaps removeLeases skips missing leases and calls a removeLease that requires a lease, and the close edit op calls removeLease? Imagine that a client gets SIGSTOP, and then its hard lease on its open-for-write file expires and is removed. The client can be resumed later, come back and then try to close. So not being able to find a lease during a close is certainly worthy of a nasty log message, but I don't think we can call it a bug. Haohui Mai : I apologize if it is a dumb question, but do we need to pass removedUCFiles around everywhere? Can't we just look at the inodes in the lists we're passing and see if they're under construction? Jing Zhao 's made a good point about how we don't want to remove leases for open files just because we deleted an under-construction inode inside a snapshot. But we should be able to tell whether or not we're not removing the "current" version of the UC inode somehow, right?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12658724/HDFS-6757.004.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestLeaseRecovery2

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7502//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7502//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658724/HDFS-6757.004.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestLeaseRecovery2 +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7502//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7502//console This message is automatically generated.
        Hide
        Jing Zhao added a comment -

        It might not be sufficient as previous implementation of snapshot might replace an inode with its subclass

        With INode features we now no longer have INode replacement (exception for INodeReference which is out of picture in this jira), thus I think it's fine to use INode in the lease map. However, in the meanwhile, I'm not sure if this will increase the possibility of memory leak due to failing to update the lease map (maybe because of buggy code).

        In serialization's special handling of UC snapshot files

        Actually this part of code (I guess you mean the special handling in saveFilesUnderConstruction) is no longer called by the normal code path after we use Protobuf for fsimage (only called by saveLegacyOIVImage). Initially this part of code is a temporary solution for HDFS-5428.

        Show
        Jing Zhao added a comment - It might not be sufficient as previous implementation of snapshot might replace an inode with its subclass With INode features we now no longer have INode replacement (exception for INodeReference which is out of picture in this jira), thus I think it's fine to use INode in the lease map. However, in the meanwhile, I'm not sure if this will increase the possibility of memory leak due to failing to update the lease map (maybe because of buggy code). In serialization's special handling of UC snapshot files Actually this part of code (I guess you mean the special handling in saveFilesUnderConstruction ) is no longer called by the normal code path after we use Protobuf for fsimage (only called by saveLegacyOIVImage ). Initially this part of code is a temporary solution for HDFS-5428 .
        Hide
        Haohui Mai added a comment -

        The v4 patch addresses both Daryn Sharp and Colin Patrick McCabe's comments. The main differences between the v3 and v4 patch is to add a new parameter, removedUCFiles, to track the leases that have to be revoked during deletes and renames.

        Show
        Haohui Mai added a comment - The v4 patch addresses both Daryn Sharp and Colin Patrick McCabe 's comments. The main differences between the v3 and v4 patch is to add a new parameter, removedUCFiles , to track the leases that have to be revoked during deletes and renames.
        Hide
        Haohui Mai added a comment -

        Thanks Daryn Sharp for the review. I'm updating my patch to address the comments. Here are some of my thoughts:

        Why not track inodes directly instead of via Longs? It will avoid unnecessary lookups in multiple places.

        It might not be sufficient as previous implementation of snapshot might replace an inode with its subclass (e.g., replace INodeDirectory to INodeDirectoryWithSnapshots). I'm not sure whether the behavior still exists today, I think Jing Zhao might have a better idea on whether this is okay. Jing Zhao, can you comment on this?

        However, isn't a bug if there's no lease during a close op? Perhaps removeLeases skips missing leases and calls a removeLease that requires a lease, and the close edit op calls removeLease?

        FSImageFormatPBINode#serializeFileUCSection skips non-existent inodes reported from the lease manager. When would this happen other than due to bug that corrupted the lease manager?

        I think that your points are quite fair. My thinking is that even though the lease manager is buggy, NN should continue to function when replay edit logs / saving namespaces instead of crashing. Though not ideal, but it allows NN to tolerate the bugs and to treat them as lease recoveries. Please see TestSaveNamespace#testSaveNamespaceWithDanglingLease for more details.

        I'm updating my patch to print out warnings to help developers to catch these bugs.

        In serialization's special handling of UC snapshot files: Is this for all files in the mutable snapshotted directory, or immutable files only in a snapshot? If the latter, isn't it a bug that the lease wasn't already revoked?

        I think that the use case is that the user takes a snapshot which contains an opened file, and then deletes the file subsequently. Maybe Jing Zhao will have a better idea on this.

        My concern is how the current NN will react to a dangling lease on the origin of the renamed path. What happens when it saves the image, checkpoints, applies edits for subsequent creates of the origin path, when another client tries to recreate the origin path, etc?

        You're right that the new lease manager won't (and doesn't need to) update the lease during the renames, but I'm not sure that I fully understand your comments. Are you concerned about the new lease manager will mix the leases between the original and the renamed paths? Based on my understanding it should not happen because inode ids are unique, but I might misunderstand your comments so please feel free to provide more details.

        Show
        Haohui Mai added a comment - Thanks Daryn Sharp for the review. I'm updating my patch to address the comments. Here are some of my thoughts: Why not track inodes directly instead of via Longs? It will avoid unnecessary lookups in multiple places. It might not be sufficient as previous implementation of snapshot might replace an inode with its subclass (e.g., replace INodeDirectory to INodeDirectoryWithSnapshots ). I'm not sure whether the behavior still exists today, I think Jing Zhao might have a better idea on whether this is okay. Jing Zhao , can you comment on this? However, isn't a bug if there's no lease during a close op? Perhaps removeLeases skips missing leases and calls a removeLease that requires a lease, and the close edit op calls removeLease? FSImageFormatPBINode#serializeFileUCSection skips non-existent inodes reported from the lease manager. When would this happen other than due to bug that corrupted the lease manager? I think that your points are quite fair. My thinking is that even though the lease manager is buggy, NN should continue to function when replay edit logs / saving namespaces instead of crashing. Though not ideal, but it allows NN to tolerate the bugs and to treat them as lease recoveries. Please see TestSaveNamespace#testSaveNamespaceWithDanglingLease for more details. I'm updating my patch to print out warnings to help developers to catch these bugs. In serialization's special handling of UC snapshot files: Is this for all files in the mutable snapshotted directory, or immutable files only in a snapshot? If the latter, isn't it a bug that the lease wasn't already revoked? I think that the use case is that the user takes a snapshot which contains an opened file, and then deletes the file subsequently. Maybe Jing Zhao will have a better idea on this. My concern is how the current NN will react to a dangling lease on the origin of the renamed path. What happens when it saves the image, checkpoints, applies edits for subsequent creates of the origin path, when another client tries to recreate the origin path, etc? You're right that the new lease manager won't (and doesn't need to) update the lease during the renames, but I'm not sure that I fully understand your comments. Are you concerned about the new lease manager will mix the leases between the original and the renamed paths? Based on my understanding it should not happen because inode ids are unique, but I might misunderstand your comments so please feel free to provide more details.
        Hide
        Daryn Sharp added a comment -

        I like LeaseManager tracking inodes rather than paths! It's something Kihwal and I have long been meaning to do. I have two primary areas of concern. First, we need to ensure that error conditions from bugs are not silently tolerated. I've cited them in the following list. Second, that we haven't created an incompatibility, discusses at the end.

        1. Why not track inodes directly instead of via Longs? It will avoid unnecessary lookups in multiple places.
        2. Minor, FSEditLogLoader#applyEditLogOp has a line that overflows 80-chars.
        3. FSEditLogLoader#applyEditLogOp for a close op constructs a list so it can call LeaseManager#removeLeases. That skips non-existent leases because it's also invoked by FSNamesystem#removePathAndBlocks during deletes and not all removed inodes may have a lease. However, isn't a bug if there's no lease during a close op? Perhaps removeLeases skips missing leases and calls a removeLease that requires a lease, and the close edit op calls removeLease?
        4. In serialization's special handling of UC snapshot files: Is this for all files in the mutable snapshotted directory, or immutable files only in a snapshot? If the latter, isn't it a bug that the lease wasn't already revoked?
        5. In FSNamesystem#getCompleteBlocksTotal, please add spaces after ifs and curlies around the 1-statement bodies. I know it's just unindented existing code but it should be consistent with the coding standards.
        6. FSImageFormatPBINode#serializeFileUCSection skips non-existent inodes reported from the lease manager. When would this happen other than due to bug that corrupted the lease manager?
        7. LeaseManager#getPaths used to return Collection but when renamed to getFiles it returns HashSet. It would be better to either retain Collection or use generic Set for the signature.

        A possible compatibility issue due to a subtle functionality change: Renames no longer reassign a lease which I think makes sense going forward since we aren't tracking paths anymore. However, will backing up to a prior hdfs version (with the same image format) have issues? I haven't traced the code due to time constraints. My concern is how the current NN will react to a dangling lease on the origin of the renamed path. What happens when it saves the image, checkpoints, applies edits for subsequent creates of the origin path, when another client tries to recreate the origin path, etc?

        Show
        Daryn Sharp added a comment - I like LeaseManager tracking inodes rather than paths! It's something Kihwal and I have long been meaning to do. I have two primary areas of concern. First, we need to ensure that error conditions from bugs are not silently tolerated. I've cited them in the following list. Second, that we haven't created an incompatibility, discusses at the end. Why not track inodes directly instead of via Longs? It will avoid unnecessary lookups in multiple places. Minor, FSEditLogLoader#applyEditLogOp has a line that overflows 80-chars. FSEditLogLoader#applyEditLogOp for a close op constructs a list so it can call LeaseManager#removeLeases . That skips non-existent leases because it's also invoked by FSNamesystem#removePathAndBlocks during deletes and not all removed inodes may have a lease. However, isn't a bug if there's no lease during a close op? Perhaps removeLeases skips missing leases and calls a removeLease that requires a lease, and the close edit op calls removeLease ? In serialization's special handling of UC snapshot files: Is this for all files in the mutable snapshotted directory, or immutable files only in a snapshot? If the latter, isn't it a bug that the lease wasn't already revoked? In FSNamesystem#getCompleteBlocksTotal , please add spaces after ifs and curlies around the 1-statement bodies. I know it's just unindented existing code but it should be consistent with the coding standards. FSImageFormatPBINode#serializeFileUCSection skips non-existent inodes reported from the lease manager. When would this happen other than due to bug that corrupted the lease manager? LeaseManager#getPaths used to return Collection but when renamed to getFiles it returns HashSet . It would be better to either retain Collection or use generic Set for the signature. A possible compatibility issue due to a subtle functionality change: Renames no longer reassign a lease which I think makes sense going forward since we aren't tracking paths anymore. However, will backing up to a prior hdfs version (with the same image format) have issues? I haven't traced the code due to time constraints. My concern is how the current NN will react to a dangling lease on the origin of the renamed path. What happens when it saves the image, checkpoints, applies edits for subsequent creates of the origin path, when another client tries to recreate the origin path, etc?
        Hide
        Daryn Sharp added a comment -

        I'm currently reviewing.

        Show
        Daryn Sharp added a comment - I'm currently reviewing.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12658259/HDFS-6757.003.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7478//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7478//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658259/HDFS-6757.003.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7478//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7478//console This message is automatically generated.
        Hide
        Jing Zhao added a comment -

        The patch looks good to me. One issue is that the current patch cannot handle the scenario where an under construction file, which is contained in a snapshot, is deleted. The original code still calls removeLeaseWithPrefixPath in removePathAndBlocks, thus even if removedINodes is empty or does not contain UC files that are in snapshots the corresponding leases are still updated. Now only updating the leaseMap based on removedINodes cannot update these leases.

        Maybe we can use another list to collect all the UC files in cleanSubtree. But we need to make sure all the files are touched by cleanSubtree and more unit tests need to be added.

        Besides, some minors and nits:

        1. Considering getFullPathName is a heavy recursive call, and when we call
          removeLease from finalizeINodeFileUnderConstruction we already know the
          full path, maybe here we can either pass in the full path as a parameter?
          -  synchronized void removeLease(String holder, String src) {
          +  synchronized void removeLease(String holder, INodeFile src) {
               Lease lease = getLease(holder);
               if (lease != null) {
          -      removeLease(lease, src);
          +      removeLease(lease, src.getId());
               } else {
                 LOG.warn("Removing non-existent lease! holder=" + holder +
          -          " src=" + src);
          +          " src=" + src.getFullPathName());
               }
             }
          
        2. The following code needs some clean:
          +              LOG.debug("Lease recovery for inode " + id + " is complete. " +
          +                      "File" +
          +                      " closed.");
          
        -    ((Log4JLogger)DataNode.LOG).getLogger().setLevel(Level.ALL);
        +    //((Log4JLogger)DataNode.LOG).getLogger().setLevel(Level.ALL);
        
        Show
        Jing Zhao added a comment - The patch looks good to me. One issue is that the current patch cannot handle the scenario where an under construction file, which is contained in a snapshot, is deleted. The original code still calls removeLeaseWithPrefixPath in removePathAndBlocks , thus even if removedINodes is empty or does not contain UC files that are in snapshots the corresponding leases are still updated. Now only updating the leaseMap based on removedINodes cannot update these leases. Maybe we can use another list to collect all the UC files in cleanSubtree . But we need to make sure all the files are touched by cleanSubtree and more unit tests need to be added. Besides, some minors and nits: Considering getFullPathName is a heavy recursive call, and when we call removeLease from finalizeINodeFileUnderConstruction we already know the full path, maybe here we can either pass in the full path as a parameter? - synchronized void removeLease( String holder, String src) { + synchronized void removeLease( String holder, INodeFile src) { Lease lease = getLease(holder); if (lease != null ) { - removeLease(lease, src); + removeLease(lease, src.getId()); } else { LOG.warn( "Removing non-existent lease! holder=" + holder + - " src=" + src); + " src=" + src.getFullPathName()); } } The following code needs some clean: + LOG.debug( "Lease recovery for inode " + id + " is complete. " + + "File" + + " closed." ); - ((Log4JLogger)DataNode.LOG).getLogger().setLevel(Level.ALL); + //((Log4JLogger)DataNode.LOG).getLogger().setLevel(Level.ALL);
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12658200/HDFS-6757.002.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7477//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7477//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658200/HDFS-6757.002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7477//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7477//console This message is automatically generated.
        Hide
        Haohui Mai added a comment -

        Filed HDFS-6765 for the follow-up issue.

        Show
        Haohui Mai added a comment - Filed HDFS-6765 for the follow-up issue.
        Hide
        Brandon Li added a comment -

        When INode ID was introduced a year ago, one concern was the debug could be a bit more difficult. For example, the following trace may not help debug that much when error happens.

        -        LOG.debug(src + " not found in lease.paths (=" + lease.paths + ")");
        +        LOG.debug("inode " + inodeId + " not found in lease.files (=" + lease
        +                + ")"); 
        

        It can be helpful to have the full path logged once or two in some places.

        Show
        Brandon Li added a comment - When INode ID was introduced a year ago, one concern was the debug could be a bit more difficult. For example, the following trace may not help debug that much when error happens. - LOG.debug(src + " not found in lease.paths (=" + lease.paths + ")"); + LOG.debug("inode " + inodeId + " not found in lease.files (=" + lease + + ")"); It can be helpful to have the full path logged once or two in some places.
        Hide
        Colin Patrick McCabe added a comment -

        Agree. However, the issue here is that internalReleaseLease eventually calls closeFile, which logs the full path in the edit log. We probably need to change closeFile to log inodeids instead of full paths before making the proposed change. Do you think whether it is okay to address it in a follow-up jira?

        I think a follow-up is fine.

        Show
        Colin Patrick McCabe added a comment - Agree. However, the issue here is that internalReleaseLease eventually calls closeFile, which logs the full path in the edit log. We probably need to change closeFile to log inodeids instead of full paths before making the proposed change. Do you think whether it is okay to address it in a follow-up jira? I think a follow-up is fine.
        Hide
        Haohui Mai added a comment -

        Thanks Colin for the review.

        ... Perhaps we should just make internalReleaseLease take an INode ID?

        Agree. However, the issue here is that internalReleaseLease eventually calls closeFile, which logs the full path in the edit log. We probably need to change closeFile to log inodeids instead of full paths before making the proposed change. Do you think whether it is okay to address it in a follow-up jira?

        Show
        Haohui Mai added a comment - Thanks Colin for the review. ... Perhaps we should just make internalReleaseLease take an INode ID? Agree. However, the issue here is that internalReleaseLease eventually calls closeFile , which logs the full path in the edit log. We probably need to change closeFile to log inodeids instead of full paths before making the proposed change. Do you think whether it is okay to address it in a follow-up jira?
        Hide
        Colin Patrick McCabe added a comment -

        This JIRA seems like a good simplification.

             /** Does this lease contain any path? */
        -    boolean hasPath() {return !paths.isEmpty();}
        +    boolean hasPath() {return !files.isEmpty();}
        

        Seems like this should be renamed to 'hasFile' or something like that, now that class Lease no longer contains paths.

        Also, Lease no longer needs to implement Comparable since we're using hash sets now, right?

        +      // need to create a copy of the oldest lease files, becuase
        

        typo

        +          boolean completed = fsnamesystem.internalReleaseLease(oldest,
        +                  fsnamesystem.getFSDirectory().getInode(id).getFullPathName(),
        

        Here we're taking an inode ID and then translating it to an INode, then translating that INode to a path, then passing it to a function which will need to do a lookup on that path to get the INode ID back. Perhaps we should just make internalReleaseLease take an INode ID?

        Show
        Colin Patrick McCabe added a comment - This JIRA seems like a good simplification. /** Does this lease contain any path? */ - boolean hasPath() { return !paths.isEmpty();} + boolean hasPath() { return !files.isEmpty();} Seems like this should be renamed to 'hasFile' or something like that, now that class Lease no longer contains paths. Also, Lease no longer needs to implement Comparable since we're using hash sets now, right? + // need to create a copy of the oldest lease files, becuase typo + boolean completed = fsnamesystem.internalReleaseLease(oldest, + fsnamesystem.getFSDirectory().getInode(id).getFullPathName(), Here we're taking an inode ID and then translating it to an INode, then translating that INode to a path, then passing it to a function which will need to do a lookup on that path to get the INode ID back. Perhaps we should just make internalReleaseLease take an INode ID?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12657925/HDFS-6757.000.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
        org.apache.hadoop.hdfs.TestLeaseRecovery2
        org.apache.hadoop.hdfs.server.balancer.TestBalancer
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
        org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
        org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7469//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7469//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657925/HDFS-6757.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7469//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7469//console This message is automatically generated.

          People

          • Assignee:
            Haohui Mai
            Reporter:
            Haohui Mai
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development