[OAK-7262] LockBasedScheduler#getHeadNodeState poor performance due to lock contention in commitTimeHistogram implementation

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0, 1.10.0, 1.8.3
• Component/s: segment-tar
• Assignee: Andrei Dulceanu
• Reporter: Andrei Dulceanu
• Created: 13/Feb/18 13:35
• Updated: 04/Oct/19 17:52
• Resolved: 16/Feb/18 08:36

Description

With OAK-4732, we slightly prioritised reads, allowing callers of LockBasedScheduler#getHeadNodeState to wait for a certain amount of time on the commit semaphore, in order to fetch the latest head. The amount of time to wait for was computed as the median of the latest 1000 commits, but the implementation was flexible enough to replace the 0.5 quantile with a different value.

The initial implementation used SynchronizedDescriptiveStatistics from commons-math3. In OAK-6430, we replaced that with a Histogram backed by a SlindingWindowReservoir from the dropwizard metric library (io.dropwizard.metrics). 

One of the problems observed with the current implementation is that, under heavy loading, the performance of LockBasedScheduler#getHeadNodeState decreases due to lock contention in commitTimeHistogram implementation. Namely,  SlidingWindowReservoir#getSnapshot seems to be a bottleneck. It copies an array of 1000 elements at each invocation of LockBasedScheduler#getHeadNodeState, acquiring and releasing a lock for each element in the array.