Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2802 Support for RW/RO snapshots in HDFS
  3. HDFS-4749

Use INodeId to identify the corresponding directory node for FSImage saving/loading

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Snapshot (HDFS-2802)
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently in fsimage, we use the path to locate a directory node for later loading, i.e., when loading a subtree from fsimage, we first read the path of the directory node, and resolve the path to identify the directory node. This brings extra complexity since we need to generate path for directory nodes in both the current tree and snapshot copies.

      As a simplification, we can use INodeId to identify the directory node.

        Activity

        Jing Zhao created issue -
        Jing Zhao made changes -
        Field Original Value New Value
        Component/s datanode [ 12312927 ]
        Component/s namenode [ 12312926 ]
        Hide
        Jing Zhao added a comment -

        Another issue of using path is that it cannot handle (or it's very difficult to handle) the fsimage after the following operations:

        1. Create directory /dir1, /dir2, /dir1/foo, and /dir1/foo/bar
        2. Allow and make snapshot s1 on /dir1
        3. Rename /dir1/foo/bar to /dir2/bar
        4. Rename /dir1/foo to /dir2/bar/foo

        The current implementation will write a path /dir1/.snapshot/s1/foo/bar/foo to the fsimage, which cannot be correctly resolved.

        Show
        Jing Zhao added a comment - Another issue of using path is that it cannot handle (or it's very difficult to handle) the fsimage after the following operations: 1. Create directory /dir1, /dir2, /dir1/foo, and /dir1/foo/bar 2. Allow and make snapshot s1 on /dir1 3. Rename /dir1/foo/bar to /dir2/bar 4. Rename /dir1/foo to /dir2/bar/foo The current implementation will write a path /dir1/.snapshot/s1/foo/bar/foo to the fsimage, which cannot be correctly resolved.
        Hide
        Jing Zhao added a comment -

        Initial patch.

        Show
        Jing Zhao added a comment - Initial patch.
        Jing Zhao made changes -
        Attachment HDFS-4749.000.patch [ 12580430 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good. This cleans up a lot of code.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good. This cleans up a lot of code.
        Tsz Wo Nicholas Sze made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Component/s datanode [ 12312927 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Jing!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Jing!
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s Snapshot (HDFS-2802) [ 12323386 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-Snapshots-Branch-build #169 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/169/)
        HDFS-4749. Use INodeId to identify the corresponding directory node in FSImage saving/loading. Contributed by Jing Zhao (Revision 1475902)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475902
        Files :

        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #169 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/169/ ) HDFS-4749 . Use INodeId to identify the corresponding directory node in FSImage saving/loading. Contributed by Jing Zhao (Revision 1475902) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475902 Files : /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES. HDFS-2802 .txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java

          People

          • Assignee:
            Jing Zhao
            Reporter:
            Jing Zhao
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development