[HDFS-9833] Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha1
• Labels: hdfs-ec-3.0-must-do
• Target Version/s: 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Rakesh Radhakrishnan
• Reporter: Kai Zheng
• Created: 19/Feb/16 06:34
• Updated: 01/Jun/16 05:00
• Resolved: 01/Jun/16 05:00

Description

As discussed in HDFS-8430 and HDFS-9694, to compute striped file checksum even some of striped blocks are missed, we need to consider recomputing block checksum on the fly for the missed/corrupt blocks. To recompute the block checksum, the block data needs to be reconstructed by erasure decoding, and the main needed codes for the block reconstruction could be borrowed from HDFS-9719, the refactoring of the existing ErasureCodingWorker. In EC worker, reconstructed blocks need to be written out to target datanodes, but here in this case, the remote writing isn't necessary, as the reconstructed block data is only used to recompute the checksum.

Issue Links

• depends upon: HDFS-9694 Make existing DFSClient#getFileChecksum() work for striped blocks
• Is contained by: HDFS-8430 Erasure coding: compute file checksum for striped files (stripe by stripe)
• relates to: HDFS-10461 Erasure Coding: Optimize block checksum recalculation logic on the fly by reconstructing multiple missed blocks at a time
• relates to: HDFS-10460 Erasure Coding: Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block