[FLINK-13013] Make sure that SingleInputGate can always request partitions

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Network
• Labels: pull-request-available

Release Note

In case when partitions are not (yet) found, because we are requesting partitions sooner (now we are requesting partitions before initialise operator states), partitions request re-triggering will also happen sooner, so the incremental back of counter will start counting also sooner.

It is possible that some Jobs (with long deployment times and large state) might start failing more frequently with {{PartitionNotFound}} exceptions compared to Flink 1.8. If that's the case users should increase {{taskmanager.network.request-backoff.max}} value in order to have the same effective request partitions time out as it was in Flink 1.8.

• Assignee: Piotr Nowojski
• Reporter: Piotr Nowojski
• Created: 27/Jun/19 10:00
• Updated: 16/Jul/19 02:24
• Resolved: 09/Jul/19 12:54

Description

Currently SingleInputGate requests partitions only on the first attempt to fetch the data. Before requesting partitions, no data availability notifications can get through. This doesn't work well with a new non blocking InputGate interface, since on newly created SingleInputGates, InputGate#isAvailable() might return not available, and it will be only able to switch to available, after first call to SingleInputGate#pollNext(). However this might never happen, since caller could be waiting indefinitely on SingleInputGate#isAvailable().

Issue Links

• links to: GitHub Pull Request #9017