Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1
    • Fix Version/s: 0.18.4, 0.19.2, 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Restarting datanodes while a client is writing to it can cause namenode to get stuck in safe mode.

      1. 5644.trunk.patch
        0.9 kB
        Suresh Srinivas
      2. 5644.trunk.patch
        0.9 kB
        Suresh Srinivas
      3. 5644.branch-0.19.patch
        0.8 kB
        Suresh Srinivas
      4. 5644.branch-0.19.patch
        0.8 kB
        Suresh Srinivas
      5. 5644.branch-0.19.patch
        0.7 kB
        Suresh Srinivas
      6. 5644.branch-0.18.patch
        0.7 kB
        Suresh Srinivas
      7. 5644.branch-0.18.patch
        0.7 kB
        Suresh Srinivas
      8. 5644.branch-0.18.patch
        0.7 kB
        Suresh Srinivas

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #811 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/811/ )
        Hide
        Hairong Kuang added a comment -

        I just committed this. Thanks, Suresh!

        Show
        Hairong Kuang added a comment - I just committed this. Thanks, Suresh!
        Hide
        Suresh Srinivas added a comment -

        The failed tests are not related to this change.

        This functionality is tested manually as it is not easy to automate this test. To test this patch run a command show below:

        1. Set dfs.safemode.threshld.pct to 1.000
        2. Run slowout | bin/hadoop fs -put - /test.out
          where slowout is a command that outputs text slowly. While the output of the command is being written to ~/test.out, kill the datanode to which client is writing to, followed by the other datanodes to which the file is being pipelined to.
        3. Restart the cluster. The cluster gets stuck in safe mode without this patch. With this patch, cluster starts up and automatically comes out of safe mode.
        Show
        Suresh Srinivas added a comment - The failed tests are not related to this change. This functionality is tested manually as it is not easy to automate this test. To test this patch run a command show below: Set dfs.safemode.threshld.pct to 1.000 Run slowout | bin/hadoop fs -put - /test.out where slowout is a command that outputs text slowly. While the output of the command is being written to ~/test.out, kill the datanode to which client is writing to, followed by the other datanodes to which the file is being pipelined to. Restart the cluster. The cluster gets stuck in safe mode without this patch. With this patch, cluster starts up and automatically comes out of safe mode.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12405204/5644.trunk.patch
        against trunk revision 764287.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405204/5644.trunk.patch against trunk revision 764287. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Attaching the patch for the trunk as the latest attachment.

        Show
        Suresh Srinivas added a comment - Attaching the patch for the trunk as the latest attachment.
        Hide
        Suresh Srinivas added a comment -

        Thanks Nicholas. Attaching the right files this time

        Show
        Suresh Srinivas added a comment - Thanks Nicholas. Attaching the right files this time
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Good observation!

        Your current 0.18 and 0.19 patches still remove the log messages. I guess you have uploaded wrong files.

        Show
        Tsz Wo Nicholas Sze added a comment - Good observation! Your current 0.18 and 0.19 patches still remove the log messages. I guess you have uploaded wrong files.
        Hide
        Hairong Kuang added a comment -

        +1. The patch looks good.

        Show
        Hairong Kuang added a comment - +1. The patch looks good.
        Hide
        Suresh Srinivas added a comment -

        Adding LOG that was deleted in the patch versions for 18 and 19

        Show
        Suresh Srinivas added a comment - Adding LOG that was deleted in the patch versions for 18 and 19
        Hide
        Suresh Srinivas added a comment -
        1. For branches 20 and trunk the block is not persisted in FSNamesystem.commitBlockSynchronization(), if append support is not enabled. Trunk patch can be applied to branch 20.
        2. For branches 18 and 19, since there is no flag to enable/disable support for append, block is not persisted without any check for append support.

        This needs to be revisited when append functionality is enabled (that is, even with append/sync, if a block is persisted, the datanode must not delete the block from /tmp).

        I have tested this fix manually on trunk.

        Show
        Suresh Srinivas added a comment - For branches 20 and trunk the block is not persisted in FSNamesystem.commitBlockSynchronization() , if append support is not enabled. Trunk patch can be applied to branch 20. For branches 18 and 19, since there is no flag to enable/disable support for append, block is not persisted without any check for append support. This needs to be revisited when append functionality is enabled (that is, even with append/sync, if a block is persisted, the datanode must not delete the block from /tmp). I have tested this fix manually on trunk.
        Hide
        Hairong Kuang added a comment -

        +1. CommitBlockSynchronization() should not persist blocks if sync/append is not supported.

        Show
        Hairong Kuang added a comment - +1. CommitBlockSynchronization() should not persist blocks if sync/append is not supported.
        Hide
        Suresh Srinivas added a comment -

        Proposed solution:
        When a client encounters a problem, it tries to recover the block. During block recovery, FSNamesystem.commitBlockSynchronization() the information related to the block is persisted. Namenode during restart, loads the information related to the failed block. The solution is not to persist the block synchronization during block recovery.

        Show
        Suresh Srinivas added a comment - Proposed solution: When a client encounters a problem, it tries to recover the block. During block recovery, FSNamesystem.commitBlockSynchronization() the information related to the block is persisted. Namenode during restart, loads the information related to the failed block. The solution is not to persist the block synchronization during block recovery.
        Hide
        Suresh Srinivas added a comment -

        A client is writing data to datanode D1, with pipeline to datanodes D2 and D3. Restarting D1, followed by D2 and D3 results in:
        1. File that client was writing gets stuck in the state under construction. Restarting the cluster or lease recovery timeout does not finalize the file.
        2. Block that client was writing to is deleted by the datanodes during restart. If the namenode is configured with safemode threshold of 1.0, on restart it does not come out of safemode, since the deleted block is never reported by any datanode.

        Show
        Suresh Srinivas added a comment - A client is writing data to datanode D1, with pipeline to datanodes D2 and D3. Restarting D1, followed by D2 and D3 results in: 1. File that client was writing gets stuck in the state under construction. Restarting the cluster or lease recovery timeout does not finalize the file. 2. Block that client was writing to is deleted by the datanodes during restart. If the namenode is configured with safemode threshold of 1.0, on restart it does not come out of safemode, since the deleted block is never reported by any datanode.

          People

          • Assignee:
            Suresh Srinivas
            Reporter:
            Suresh Srinivas
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development