Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2802 Support for RW/RO snapshots in HDFS
  3. HDFS-4749

Use INodeId to identify the corresponding directory node for FSImage saving/loading

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Snapshot (HDFS-2802)
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently in fsimage, we use the path to locate a directory node for later loading, i.e., when loading a subtree from fsimage, we first read the path of the directory node, and resolve the path to identify the directory node. This brings extra complexity since we need to generate path for directory nodes in both the current tree and snapshot copies.

      As a simplification, we can use INodeId to identify the directory node.

        Activity

        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-Snapshots-Branch-build #169 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/169/)
        HDFS-4749. Use INodeId to identify the corresponding directory node in FSImage saving/loading. Contributed by Jing Zhao (Revision 1475902)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475902
        Files :

        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #169 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/169/ ) HDFS-4749 . Use INodeId to identify the corresponding directory node in FSImage saving/loading. Contributed by Jing Zhao (Revision 1475902) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475902 Files : /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES. HDFS-2802 .txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Jing!

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Jing!
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good. This cleans up a lot of code.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - +1 patch looks good. This cleans up a lot of code.
        Hide
        jingzhao Jing Zhao added a comment -

        Initial patch.

        Show
        jingzhao Jing Zhao added a comment - Initial patch.
        Hide
        jingzhao Jing Zhao added a comment -

        Another issue of using path is that it cannot handle (or it's very difficult to handle) the fsimage after the following operations:

        1. Create directory /dir1, /dir2, /dir1/foo, and /dir1/foo/bar
        2. Allow and make snapshot s1 on /dir1
        3. Rename /dir1/foo/bar to /dir2/bar
        4. Rename /dir1/foo to /dir2/bar/foo

        The current implementation will write a path /dir1/.snapshot/s1/foo/bar/foo to the fsimage, which cannot be correctly resolved.

        Show
        jingzhao Jing Zhao added a comment - Another issue of using path is that it cannot handle (or it's very difficult to handle) the fsimage after the following operations: 1. Create directory /dir1, /dir2, /dir1/foo, and /dir1/foo/bar 2. Allow and make snapshot s1 on /dir1 3. Rename /dir1/foo/bar to /dir2/bar 4. Rename /dir1/foo to /dir2/bar/foo The current implementation will write a path /dir1/.snapshot/s1/foo/bar/foo to the fsimage, which cannot be correctly resolved.

          People

          • Assignee:
            jingzhao Jing Zhao
            Reporter:
            jingzhao Jing Zhao
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development