[IGNITE-20373] Fix IO threading model

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0
• Fix Version/s: 3.0
• Labels: ignite-3 ignite3_performance threading
• Assignee: Roman Puchkovskiy
• Reporter: Alexey Scherbakov
• Reviewer: Vladislav Pyatkov
• Created: 07/Sep/23 07:51
• Updated: 04/Sep/24 16:10
• Resolved: 29/Feb/24 11:31

Description

Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread)

It makes message processing essentially single threaded and introduces additional lag due to message transition to additional queue.

addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft.

inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor. Delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A. A stripe is defined by sender-receiver pair (or can be user defined - TBD)

outboundExecutor also looks like a contention point - need to be addressed as well

Issue Links

• blocks: IGNITE-21541 Avoid partition-operations pool when it not lead to starvation
• is blocked by: IGNITE-21601 Avoid watch thread hijack due to schema sync in KV/record views
• relates to: IGNITE-21458 Up to 25% performance drop in key-value operations (embedded node)
• relates to: IGNITE-21600 Make partition-operations a plain executor
• split to: IGNITE-21627 Make Outbound pool striped
• split to: IGNITE-21626 Make each component of the Inbound pool striped
• links to: GitHub Pull Request #3300