Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
One motivation of this issue is for reducing the in-flight data in the case of back pressure to speed up checkpoint. The current default exclusive buffers per channel is 2. If we reduce it to 0 and increase somewhat floating buffers for compensation, it might cause deadlock problem because all the floating buffers might be requested away by some blocked input channels and never recycled until barrier alignment.
In order to solve above deadlock concern, we can make some logic changes on both sender and receiver sides.
- Sender side: It should revoke previous received credit after sending checkpoint barrier, that means it would not send any following buffers until receiving new credits.
- Receiver side: The respective channel releases the requested floating buffers if barrier is received from the network. After barrier alignment, it would request floating buffers for the channels with positive backlog, and notify the sender side of available credits. Then the sender can continue transporting the buffers.
Based on above changes, we can also remove the `BufferStorage` component completely, because the receiver would never reading buffers for blocked channels. Another possible benefit is that the floating buffers might be more properly made use of before barrier alignment.
The only side effect would bring somehow cold setup after barrier alignment. That means the sender side has to wait for credit feedback to transport data just after alignment, which would impact on delay and network throughput. But considering the checkpoint interval not too short in general, so the above side effect can be ignored in practice. We can further verify it via existing micro-benchmark.
After this ticket done, we still can not set exclusive buffers to zero ATM, there exists another deadlock issue which would be solved separately in another ticket.
Attachments
Attachments
Issue Links
- causes
-
FLINK-17477 resumeConsumption call should happen as quickly as possible to minimise latency
- Reopened
- relates to
-
FLINK-17572 Remove checkpoint alignment buffered metric from webui
- Closed
- links to