Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Snapshot (HDFS-2802)
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After a long running stress test for snapshots, disallow snapshot throws IllegalStateException

      -bash-4.1$ hadoop dfs -rmr -skipTrash /user/foo/hdfs-snapshots
      rmr: cannot remove /user/foo/hdfs-snapshots: No such file or directory.

      -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfs -mkdir /user/foo/hdfs-snapshots/test2

      -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfsadmin -allowSnapshot /user/foo/hdfs-snapshots/test2
      Allowing snaphot on /user/foo/hdfs-snapshots/test2 succeeded

      -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfsadmin -disallowSnapshot /user/foo/hdfs-snapshots/test2
      disallowSnapshot: java.io.IOException: java.lang.IllegalStateException: this is already an INodeDirectoryWithSnapshot, this=test2

      1. HDFS-4735.000.patch
        0.9 kB
        Jing Zhao
      2. HDFS-4735.001.patch
        12 kB
        Jing Zhao

        Activity

        Ramya Sunil created issue -
        Ramya Sunil made changes -
        Field Original Value New Value
        Component/s datanode [ 12312927 ]
        Component/s namenode [ 12312926 ]
        Ramya Sunil made changes -
        Component/s namenode [ 12312926 ]
        Component/s datanode [ 12312927 ]
        Hide
        Ramya Sunil added a comment -

        Logs from the namenode:

        INFO org.apache.hadoop.ipc.Server: IPC Server handler 31 on 8020, call disallowSnapshot(/user/foo/hdfs-snapshots/test2) from .*: error: java.io.IOException: java.lang.IllegalStateException: this is already an INodeDirectoryWithSnapshot, this=test2
        java.io.IOException: java.lang.IllegalStateException: this is already an INodeDirectoryWithSnapshot, this=test2
                at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.replaceSelf4INodeDirectoryWithSnapshot(INodeDirectory.java:201)
                at org.apache.hadoop.hdfs.server.namenode.snapshot.INodeDirectorySnapshottable.replaceSelf(INodeDirectorySnapshottable.java:447)
                at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.resetSnapshottable(SnapshotManager.java:108)
                at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.disallowSnapshot(FSNamesystem.java:6773)
                at org.apache.hadoop.hdfs.server.namenode.NameNode.disallowSnapshot(NameNode.java:1545)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
                at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405)
                at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)
        
        Show
        Ramya Sunil added a comment - Logs from the namenode: INFO org.apache.hadoop.ipc.Server: IPC Server handler 31 on 8020, call disallowSnapshot(/user/foo/hdfs-snapshots/test2) from .*: error: java.io.IOException: java.lang.IllegalStateException: this is already an INodeDirectoryWithSnapshot, this=test2 java.io.IOException: java.lang.IllegalStateException: this is already an INodeDirectoryWithSnapshot, this=test2 at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.replaceSelf4INodeDirectoryWithSnapshot(INodeDirectory.java:201) at org.apache.hadoop.hdfs.server.namenode.snapshot.INodeDirectorySnapshottable.replaceSelf(INodeDirectorySnapshottable.java:447) at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.resetSnapshottable(SnapshotManager.java:108) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.disallowSnapshot(FSNamesystem.java:6773) at org.apache.hadoop.hdfs.server.namenode.NameNode.disallowSnapshot(NameNode.java:1545) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399)
        Tsz Wo Nicholas Sze made changes -
        Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
        Tsz Wo Nicholas Sze made changes -
        Assignee Tsz Wo (Nicholas), SZE [ szetszwo ] Jing Zhao [ jingzhao ]
        Description
        After a long running stress test for snapshots, disallow snapshot throws IllegalStateException

        -bash-4.1$ hadoop dfs -rmr -skipTrash /user/foo/hdfs-snapshots
        rmr: cannot remove /user/foo/hdfs-snapshots: No such file or directory.

        -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfs -mkdir /user/foo/hdfs-snapshots/test2

        -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfsadmin -allowSnapshot /user/foo/hdfs-snapshots/test2
        Allowing snaphot on /user/foo/hdfs-snapshots/test2 succeeded

        -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfsadmin -disallowSnapshot /user/foo/hdfs-snapshots/test2
        disallowSnapshot: java.io.IOException: java.lang.IllegalStateException: this is already an INodeDirectoryWithSnapshot, this=test2
        After a long running stress test for snapshots, disallow snapshot throws IllegalStateException

        -bash-4.1$ hadoop dfs -rmr -skipTrash /user/foo/hdfs-snapshots
        rmr: cannot remove /user/foo/hdfs-snapshots: No such file or directory.

        -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfs -mkdir /user/foo/hdfs-snapshots/test2

        -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfsadmin -allowSnapshot /user/foo/hdfs-snapshots/test2
        Allowing snaphot on /user/foo/hdfs-snapshots/test2 succeeded

        -bash-4.1$ /usr/lib/hadoop/bin/hadoop dfsadmin -disallowSnapshot /user/foo/hdfs-snapshots/test2
        disallowSnapshot: java.io.IOException: java.lang.IllegalStateException: this is already an INodeDirectoryWithSnapshot, this=test2
        Component/s namenode [ 12312926 ]
        Hide
        Jing Zhao added a comment -

        When we have nested snapshottable directories and if we try to reset the snapshottable descendant back to an regular directory, we need to replace the snapshottable descendant with an INodeDirectoryWithSnapshot. In this case, we should not do the preconditions check in INodeDirectory#replaceSelf4INodeDirectoryWithSnapshot.

        Upload a simple fix first. Will add a unit test later.

        Show
        Jing Zhao added a comment - When we have nested snapshottable directories and if we try to reset the snapshottable descendant back to an regular directory, we need to replace the snapshottable descendant with an INodeDirectoryWithSnapshot. In this case, we should not do the preconditions check in INodeDirectory#replaceSelf4INodeDirectoryWithSnapshot. Upload a simple fix first. Will add a unit test later.
        Jing Zhao made changes -
        Attachment HDFS-4735.000.patch [ 12580144 ]
        Hide
        Jing Zhao added a comment -

        Add the unit test. Also add some code clean.

        Show
        Jing Zhao added a comment - Add the unit test. Also add some code clean.
        Jing Zhao made changes -
        Attachment HDFS-4735.001.patch [ 12580194 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Tsz Wo Nicholas Sze made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Jing!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Jing!
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-Snapshots-Branch-build #167 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/167/)
        HDFS-4735. DisallowSnapshot throws IllegalStateException for nested snapshottable directories. Contributed by Jing Zhao (Revision 1471214)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1471214
        Files :

        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/Diff.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestDisallowModifyROSnapshot.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java
        • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotReplication.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #167 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/167/ ) HDFS-4735 . DisallowSnapshot throws IllegalStateException for nested snapshottable directories. Contributed by Jing Zhao (Revision 1471214) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1471214 Files : /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES. HDFS-2802 .txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/Diff.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestDisallowModifyROSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotReplication.java
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        2h 45m 1 Tsz Wo Nicholas Sze 24/Apr/13 01:28

          People

          • Assignee:
            Jing Zhao
            Reporter:
            Ramya Sunil
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development