[HDFS-15150] Introduce read write lock to Datanode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0, 2.10.2, 3.2.3, 3.2.4
• Component/s: datanode
• Target Version/s: 3.3.0
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 30/Jan/20 14:41
• Updated: 14/Sep/21 07:19
• Resolved: 22/Jun/21 15:35

Description

HDFS-9668 pointed out the issues around the DN lock being a point of contention some time ago, but that Jira went in a direction of creating a new FSDataset implementation which is very risky, and activity on the Jira has stalled for a few years now. Edit: Looks like HDFS-9668 eventually went in a similar direction to what I was thinking, so I will review that Jira in more detail to see if this one is necessary.

I feel there could be significant gains by moving to a ReentrantReadWrite lock within the DN. The current implementation is simply a ReentrantLock so any locker blocks all others.

Once place I think a read lock would benefit us significantly, is when the DN is serving a lot of small blocks and there are jobs which perform a lot of reads. The start of reading any blocks right now takes the lock, but if we moved this to a read lock, many reads could do this at the same time.

The first conservative step, would be to change the current lock and then make all accesses to it obtain the write lock. That way, we should keep the current behaviour and then we can selectively move some lock accesses to the readlock in separate Jiras.

I would appreciate any thoughts on this, and also if anyone has attempted it before and found any blockers.

Issue Links

• Dependent: HDFS-15180 DataNode FsDatasetImpl Fine-Grained Locking via BlockPool.
• relates to: HDFS-15160 ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock
• relates to: HDFS-9668 Optimize the locking in FsDatasetImpl