[HDFS-15160] ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1, 3.4.0, 3.2.3
• Component/s: datanode
• Labels: pull-request-available
• Target Version/s: 3.4.0, 2.10.2, 3.2.3
• Hadoop Flags: Reviewed
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 10/Feb/20 13:10
• Updated: 27/Jan/24 03:28
• Resolved: 11/Sep/21 13:51

Description

Now we have HDFS-15150, we can start to move some DN operations to use the read lock rather than the write lock to improve concurrence. The first step is to make the changes to ReplicaMap, as many other methods make calls to it.

This Jira switches read operations against the volume map to use the readLock rather than the write lock.

Additionally, some methods make a call to replicaMap.replicas() (eg getBlockReports, getFinalizedBlocks, deepCopyReplica) and only use the result in a read only fashion, so they can also be switched to using a readLock.

Next is the directory scanner and disk balancer, which only require a read lock.

Finally (for this Jira) are various "low hanging fruit" items in BlockSender and fsdatasetImpl where is it fairly obvious they only need a read lock.

For now, I have avoided changing anything which looks too risky, as I think its better to do any larger refactoring or risky changes each in their own Jira.

Issue Links

• causes: HDFS-15457 TestFsDatasetImpl fails intermittently
• causes: HDFS-15818 Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig
• Dependent: HDFS-15180 DataNode FsDatasetImpl Fine-Grained Locking via BlockPool.
• is related to: HDFS-15150 Introduce read write lock to Datanode
• relates to: HDFS-15415 Reduce locking in Datanode DirectoryScanner
• relates to: HDFS-15000 Improve FsDatasetImpl to avoid IO operation in datasetLock
• links to: GitHub Pull Request #2813
• links to: GitHub Pull Request #3196
• links to: GitHub Pull Request #3200
• links to: GitHub Pull Request #3432