We observed a case, client closes the file after soft lease recovery already started but before namenode commitBlockSynchronization.
This leads to commitBlockSynchronization failure with error below, which requires either the file isn't closed or the last block isn't in complete state.
As a result, we will have corrupted replicas by genstamp mismatch, since datanodes may have finished block recovery with a new genstamp.
This could happen when client delays a lot on write and comes back when lease recovery already happens by write/append/truncate request from other client.
- Client #1 finishes writing a file, but hasn't closed yet.
- Client #1 doesn't renew lease for a soft lease period.
- Another client #2 appends the same file.
- Soft lease recovery begins.
- Block recovery in datanodes finishes.
- Client #1 comes back to close the file.
- Close succeeds since Client #1 still hold the lease (lease isn't revoked until close in soft recovery).
- Namenode tries to commitBlockSynchronization with error log below.
- Namenode and datanodes have different genstamp for this file, resulting in corrupted block.
Check the state of the last block when completing the file. If it's under recovery, it means lease recovery started, but namenode hasn't commitBlockSynchronization yet.
In this case, don't complete file.
2020-02-22 22:47:04,698 INFO [IPC Server handler 32 on 8020] org.apache.hadoop.hdfs.server.namenode.FSNamesystem: commitBlockSynchronization(oldBlock=BP-269461681-10.65.230.22-1554624547020:blk_2642650669_3063725879, newgenerationstamp=3063765480, newlength=262144000, newtargets=[220.127.116.11:10010, 18.104.22.168:10010, 100.101.88.162:10010], closeFile=true, deleteBlock=false) 2020-02-22 22:47:04,698 DEBUG [IPC Server handler 32 on 8020] org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Unexpected block (=BP-269461681-10.65.230.22-1554624547020:blk_2642650669_3063725879) since the file (=132269111992796228.data.637180347427616457.tmp.132269136349107823.copying) is not under construction
HDFS-10240 Race between close/recoverLease leads to missing block