A client is writing data to datanode D1, with pipeline to datanodes D2 and D3. Restarting D1, followed by D2 and D3 results in:
1. File that client was writing gets stuck in the state under construction. Restarting the cluster or lease recovery timeout does not finalize the file.
2. Block that client was writing to is deleted by the datanodes during restart. If the namenode is configured with safemode threshold of 1.0, on restart it does not come out of safemode, since the deleted block is never reported by any datanode.

Proposed solution:
When a client encounters a problem, it tries to recover the block. During block recovery, FSNamesystem.commitBlockSynchronization() the information related to the block is persisted. Namenode during restart, loads the information related to the failed block. The solution is not to persist the block synchronization during block recovery.

+1. CommitBlockSynchronization() should not persist blocks if sync/append is not supported.

1. For branches 20 and trunk the block is not persisted in FSNamesystem.commitBlockSynchronization(), if append support is not enabled. Trunk patch can be applied to branch 20.
2. For branches 18 and 19, since there is no flag to enable/disable support for append, block is not persisted without any check for append support.

This needs to be revisited when append functionality is enabled (that is, even with append/sync, if a block is persisted, the datanode must not delete the block from /tmp).

I have tested this fix manually on trunk.

Adding LOG that was deleted in the patch versions for 18 and 19

+1. The patch looks good.

Good observation!

Your current 0.18 and 0.19 patches still remove the log messages. I guess you have uploaded wrong files.

Thanks Nicholas. Attaching the right files this time

Attaching the patch for the trunk as the latest attachment.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12405204/5644.trunk.patch
against trunk revision 764287.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/185/console

This message is automatically generated.

The failed tests are not related to this change.

This functionality is tested manually as it is not easy to automate this test. To test this patch run a command show below:

1. Set dfs.safemode.threshld.pct to 1.000
2. Run slowout | bin/hadoop fs -put - /test.out
   where slowout is a command that outputs text slowly. While the output of the command is being written to ~/test.out, kill the datanode to which client is writing to, followed by the other datanodes to which the file is being pipelined to.
3. Restart the cluster. The cluster gets stuck in safe mode without this patch. With this patch, cluster starts up and automatically comes out of safe mode.

I just committed this. Thanks, Suresh!

Integrated in Hadoop-trunk #811 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/811/)