[HDDS-2324] Enhance locking mechanism in OzoneManager

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Ozone Manager
• Labels: performance
• Target Version/s: 1.5.0
• Assignee: Ritesh Shukla
• Reporter: Rajesh Balamohan
• Created: 18/Oct/19 08:01
• Updated: 16/Dec/23 23:43

Description

OM has reentrant RW lock. With 100% read or 100% write benchmarks, it works out reasonably fine. There is already a ticket to optimize the write codepath (as it incurs reading from DB for key checks).

However, when small amount of write workload (e.g 3-5 threads) is added to the running read benchmark, throughput suffers significantly. This is due to the fact that the reader threads would get blocked often.  I have observed around 10x slower throughput (i.e 100% read benchmark was running at 12,000 TPS and with couple of writer threads added to it, it goes down to 1200-1800 TPS).

1. Instead of single write lock, one option could be good to scale out the write lock depending on the number of cores available in the system and acquire relevant lock by hashing the key.

2. Another option is to explore if we can make use of StampedLocks of JDK 8.x, which scales well when multiple readers and writers are there. But it is not a reentrant lock. So need to explore whether it can be an option or not.

 

 

 

Issue Links

• is related to: HDDS-6402 Introduce hashed key path locks for OBS bucket