[FLINK-23223] When flushAlways is enabled the subpartition may lose notification of data availability

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.3, 1.14.0, 1.12.5, 1.13.2
• Fix Version/s: 1.14.0, 1.12.5, 1.13.2
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yun Gao
• Reporter: Yun Gao
• Created: 02/Jul/21 11:45
• Updated: 23/Sep/21 18:01
• Resolved: 12/Jul/21 09:55

Description

When the flushAways is enabled (namely set buffer timeout to 0), there might be cases like:

1. The subpartition emit an event which blocks the channel
2. The subpartition produce more records. However, this records would not be notified since isBlocked = true.
3. When the downstream tasks resume the subpartition later, the subpartition would only mark isBlocked to false. For local input channels although it tries to add the channel if isAvailable = true, but this check would not pass since flushRequest = false. 

One case for this issue is https://issues.apache.org/jira/browse/FLINK-22085 which uses LocalInputChannel.

Issue Links

• causes: FLINK-22085 KafkaSourceLegacyITCase hangs/fails on azure
• links to: GitHub Pull Request #16366
• links to: GitHub Pull Request #16449
• links to: GitHub Pull Request #16450