[FLINK-24035] Fix the deadlock issue caused by buffer listeners may not be notified

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.14.0, 1.15.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 28/Aug/21 02:39
• Updated: 21/Dec/21 15:45
• Resolved: 01/Sep/21 16:09

Description

The buffer listeners are not notified when the the local buffer pool receives available notification from the global pool. This may cause potential deadlock issue:

1. A LocalBufferPool is created, but there is no available buffers in the global NetworkBufferPool.
2. The LocalBufferPool registers an available buffer listener to the global NetworkBufferPool.
3. The BufferManager requests buffers from the LocalBufferPool but no buffer is available. As a result, it registers an available buffer listener to the LocalBufferPool.
4. A buffer is recycled to the global pool and the local buffer pool is notified about the available buffer.
5. The local buffer pool requests the available buffer from the global pool but the registered available buffer listener of BufferManager is not notified and it can never get a chance to be notified so deadlock occurs.

Issue Links

• causes: FLINK-25185 StreamFaultToleranceTestBase hangs on AZP
• causes: FLINK-25407 Network stack deadlock when cancellation happens during initialisation
• is related to: FLINK-13203 [proper fix] Deadlock occurs when requiring exclusive buffer for RemoteInputChannel
• is related to: FLINK-12852 [hotfix timeout] Deadlock occurs when requiring exclusive buffer for RemoteInputChannel
• relates to: FLINK-23466 UnalignedCheckpointITCase hangs on Azure
• links to: GitHub Pull Request #17030
• links to: GitHub Pull Request #17037
• links to: GitHub Pull Request #17075