[FLINK-19026] Improve threading model of Unaligner

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.12.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Arvid Heise
• Reporter: Arvid Heise
• Created: 22/Aug/20 19:00
• Updated: 22/Jun/21 14:07
• Resolved: 24/Sep/20 14:16

Description

Currently, netty threads and task thread are both accessing Unaligner. Netty thread access it indirectly through BufferReceivedListener. Thus, we need some costly synchronization and weaken the threading model of the Unaligner.

One possibility would be to introduce priority buffers to the InputGates, which overtake other buffers and also are always processable (e.g. they bypass non-blocking outputs).

Issue Links

• causes: FLINK-19391 Deadlock during partition update
• fixes: FLINK-17413 Refactor the usages of ThreadSafeUnaligner#storeNewBuffers
• links to: GitHub Pull Request #13228