[HDFS-11187] Optimize disk access for last partial chunk checksum of Finalized replica

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.0.3
• Component/s: datanode
• Target Version/s: 2.7.6
• Hadoop Flags: Reviewed
• Assignee: Gabor Bota
• Reporter: Wei-Chiu Chuang
• Created: 29/Nov/16 18:26
• Updated: 16/Aug/20 14:27
• Resolved: 21/Feb/18 07:27

Description

The patch at HDFS-11160 ensures BlockSender reads the correct version of metafile when there are concurrent writers.

However, the implementation is not optimal, because it must always read the last partial chunk checksum from disk while holding FsDatasetImpl lock for every reader. It is possible to optimize this by keeping an up-to-date version of last partial checksum in-memory and reduce disk access.

I am separating the optimization into a new jira, because maintaining the state of in-memory checksum requires a lot more work.

Issue Links

• depends upon: HDFS-11160 VolumeScanner reports write-in-progress replicas as corrupt incorrectly
• is related to: HDFS-12136 BlockSender performance regression due to volume scanner edge case
• is related to: HDFS-15000 Improve FsDatasetImpl to avoid IO operation in datasetLock
• supercedes: HDFS-11056 Concurrent append and read operations lead to checksum error