[OAK-4122] Replace the commit semaphore in the segment node store with a scheduler

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.3, 1.8.0
• Component/s: segment-tar
• Labels: operations performance scalability throughput
• Epic Link: TarMK transaction rate
• Assignee: Andrei Dulceanu
• Reporter: Michael Dürig
• Created: 11/Mar/16 14:45
• Updated: 26/Sep/19 11:41
• Resolved: 06/Apr/17 15:35

Description

SegmentNodeStore currently uses a semaphore to coordinate concurrent commits thus relying on the scheduling algorithm of that implementation and ultimately of the JVM for in what order commits are processed.

I think it would be beneficial to replace that semaphore with an explicit queue of pending commit. This would allow us to implement a proper scheduler optimising for e.g. minimal system load, maximal throughput or minimal latency etc. A scheduler could e.g. give precedence to big commits and order commits along the order of its base revisions, which would decrease the amount of work to be done in rebasing.

Issue Links

• blocks: OAK-3349 Partial compaction
• is blocked by: OAK-4464 Move FileStore.ChangeDispatcher to TarRevisions
• is depended upon by: OAK-4015 Expedite commits from the compactor
• is depended upon by: OAK-4732 (Slightly) prioritise reads over writes
• is related to: OAK-4881 Make merge semaphore in SegmentNodeStore fair by default
• relates to: OAK-4522 Improve CommitRateLimiter to optionally block some commits