Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Snapshot (HDFS-2802)
    • namenode
    • None
    • Reviewed

    Description

      With rename, an inode may be moved from one location to another location in the namespace tree. We need to take care the case that the inode is also in some snapshots.

      Attachments

        1. h4534_20130307.patch
          42 kB
          Tsz-wo Sze
        2. h4534_20130313.patch
          43 kB
          Tsz-wo Sze
        3. h4534_20130314.patch
          49 kB
          Tsz-wo Sze
        4. h4534_20130314b.patch
          50 kB
          Tsz-wo Sze
        5. h4534_20130314c.patch
          50 kB
          Tsz-wo Sze
        6. h4534_20130318.patch
          59 kB
          Tsz-wo Sze
        7. h4534_20130319.patch
          71 kB
          Tsz-wo Sze

        Issue Links

          Activity

            szetszwo Tsz-wo Sze added a comment -

            h4534_20130307.patch: add INodeReference to support rename. The patch is not yet completed.

            szetszwo Tsz-wo Sze added a comment - h4534_20130307.patch: add INodeReference to support rename. The patch is not yet completed.
            szetszwo Tsz-wo Sze added a comment -

            h4534_20130313.patch: uses reference for rename.

            szetszwo Tsz-wo Sze added a comment - h4534_20130313.patch: uses reference for rename.
            szetszwo Tsz-wo Sze added a comment -

            h4534_20130314.patch: also updates delete when the inode is a reference.

            Since the patch is getting big, I intent to have the following task in a separate JIRA:

            • change fsimage to handle reference;
            • add rename tests.
            szetszwo Tsz-wo Sze added a comment - h4534_20130314.patch: also updates delete when the inode is a reference. Since the patch is getting big, I intent to have the following task in a separate JIRA: change fsimage to handle reference; add rename tests.
            szetszwo Tsz-wo Sze added a comment -

            h4534_20130314b.patch: renames some methods/variables.

            szetszwo Tsz-wo Sze added a comment - h4534_20130314b.patch: renames some methods/variables.
            szetszwo Tsz-wo Sze added a comment -

            h4534_20130314c.patch: fixes some bugs so that it passes all snapshot related tests.

            szetszwo Tsz-wo Sze added a comment - h4534_20130314c.patch: fixes some bugs so that it passes all snapshot related tests.
            jingzhao Jing Zhao added a comment -

            The new INodeReference looks pretty good. My thoughts about the current patch:
            1. While deleting snapshots, we may have to handle INodeReference. Using the same example in the javadoc of INodeReference: in /abc, if ref(1001, foo) is in the deleted list of the very first diff, when we want to delete the correspond diff, the current code may destroy the inode bar(1000) even if it is still referred by others? Simiarly scenarios if ref(1001, foo) is processed by a Processor. We may need to check the reference number first and decide if we need to destroy the referred INode (i.e., bar(1000)).

            2. After the reference number is decrease to 0, we need to destroy the inode (bar(1000)) and delete the reference node (ref(1003)).

            3. In both unprotectedRename methods, seems that srcParent.asDirectory.replaceChild4Reference only replaces the current child and the created list, and we need to replace the inode in the deleted list?

            Minor:
            4. INodeReference.java, INodeReference#referenced can be renamed to INodeReference#referred or some other name? Similarly update the javadoc of the class.
            5. In javadoc of INodeReference, /abc will have a child ref(1001, foo) instead of ref(1001), /xyz will have a child ref(1002) instead of ref(1002, bar)?
            6. In INodeDirectory#replaceChild, can oldChild.getParentReference() return a non-null result?
            7. WithName#setLocalName will not be called in normal cases?

            jingzhao Jing Zhao added a comment - The new INodeReference looks pretty good. My thoughts about the current patch: 1. While deleting snapshots, we may have to handle INodeReference. Using the same example in the javadoc of INodeReference: in /abc, if ref(1001, foo) is in the deleted list of the very first diff, when we want to delete the correspond diff, the current code may destroy the inode bar(1000) even if it is still referred by others? Simiarly scenarios if ref(1001, foo) is processed by a Processor. We may need to check the reference number first and decide if we need to destroy the referred INode (i.e., bar(1000)). 2. After the reference number is decrease to 0, we need to destroy the inode (bar(1000)) and delete the reference node (ref(1003)). 3. In both unprotectedRename methods, seems that srcParent.asDirectory.replaceChild4Reference only replaces the current child and the created list, and we need to replace the inode in the deleted list? Minor: 4. INodeReference.java, INodeReference#referenced can be renamed to INodeReference#referred or some other name? Similarly update the javadoc of the class. 5. In javadoc of INodeReference, /abc will have a child ref(1001, foo) instead of ref(1001), /xyz will have a child ref(1002) instead of ref(1002, bar)? 6. In INodeDirectory#replaceChild, can oldChild.getParentReference() return a non-null result? 7. WithName#setLocalName will not be called in normal cases?
            szetszwo Tsz-wo Sze added a comment -

            h4534_20130318.patch: incorporates Jing's excellent review comments and renames INode.clearReferences() to INode.clear() in order to avoid confusion.

            szetszwo Tsz-wo Sze added a comment - h4534_20130318.patch: incorporates Jing's excellent review comments and renames INode.clearReferences() to INode.clear() in order to avoid confusion.
            szetszwo Tsz-wo Sze added a comment -

            h4534_20130319.patch: fixes some bugs (thanks Jing for the help) and adds Diff.ListType to reduce code duplication.

            szetszwo Tsz-wo Sze added a comment - h4534_20130319.patch: fixes some bugs (thanks Jing for the help) and adds Diff.ListType to reduce code duplication.
            jingzhao Jing Zhao added a comment -

            The latest patch looks good to me. +1 for the patch.

            jingzhao Jing Zhao added a comment - The latest patch looks good to me. +1 for the patch.
            szetszwo Tsz-wo Sze added a comment -

            I have committed this.

            szetszwo Tsz-wo Sze added a comment - I have committed this.
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-Snapshots-Branch-build #134 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/134/)
            HDFS-4534. Add INodeReference in order to support rename with snapshots. (Revision 1458164)

            Result = FAILURE
            szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458164
            Files :

            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
            • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/Diff.java
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #134 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/134/ ) HDFS-4534 . Add INodeReference in order to support rename with snapshots. (Revision 1458164) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1458164 Files : /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES. HDFS-2802 .txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeWithAdditionalFields.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/AbstractINodeDiffList.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/Diff.java

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: