Here is more information regarding this failure. The log came from the same primary datanode:
org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_1415000632081498137_954380(length=31016448), newblock=blk_1415000632081498137_989001(length=31016448), datanode=XX
org.apache.hadoop.hdfs.server.datanode.DataNode: Received block blk_1415000632081498137_989001 of size 31016448 as part of lease recovery.
It looks that pipeline recovery succeeded at the primary datanode. The new generation stamp is 989001. But the client saw this recovery as a failure and used the old generation stamp 954380 to recover the error:
WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1415000632081498137_954380 failed because recovery from primary datanode XX failed 4 times. Will retry...
This retry went on forever.