[FLINK-16641] Announce sender's backlog to solve the deadlock issue without exclusive buffers

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Zhijiang
• Created: 18/Mar/20 04:04
• Updated: 23/Sep/21 18:02
• Resolved: 12/Jul/21 15:34

Description

This is the second ingredient besides FLINK-16404 to solve the deadlock problem without exclusive buffers.

The scenario is as follows:

• The data in subpartition with positive backlog can be sent without doubt because the exclusive credits would be feedback finally.
• Without exclusive buffers, the receiver would not request floating buffers for 0 backlog. But when the new backlog is added into such subpartition, it has no way to notify the receiver side without positive credits ATM.
• So it would result in waiting for each other between receiver and sender sides to cause deadlock. The sender waits for credit to notify backlog and the receiver waits for backlog to request floating credits.

To solve the above problem, the sender needs a separate message to announce backlog sometimes besides existing `BufferResponse`. Then the receiver can get this info to request floating buffers to feedback.

The side effect brought is to increase network transport delay and throughput regression. We can measure how much it effects in existing micro-benchmark. It might probably bear this effect to get a benefit of fast checkpoint without exclusive buffers. We can give the proper explanations in respective configuration options to let users make the final decision in practice.

Issue Links

• causes: FLINK-23466 UnalignedCheckpointITCase hangs on Azure
• relates to: FLINK-23382 Performance regression on 13.07.2021
• links to: GitHub Pull Request #11877