As per the mailing thread with the heading
'Handling read failures during recovery' we found this problem.
As part of split Logs the HMaster calls Namenode recovery. The recovery is an asynchronous process.
Even though client is getting the updated block info from Namenode on first
read failure, client is discarding the new info and using the old info only
to retrieve the data from datanode. So, all the read
retries are failing. [Method parameter reassignment - Not reflected in
In HMaster code we tend to wait for 1sec. But if the recovery had some failure then split log may not happen and may lead to dataloss.
So may be we need to decide upon the actual delay that needs to be introduced once Hmaster calls NN recovery.