[FLINK-10762] Do not try to request partitions in SingleInputGate per getRecord

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.7.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Network
• Labels: stale-major
• Assignee: Unassigned
• Reporter: Nico Kruber
• Created: 02/Nov/18 12:39
• Updated: 29/Apr/21 10:14
• Resolved: 29/Apr/21 10:14

Description

Every call to SingleInputGate#getNextBufferOrEvent() will also try to request partitions via SingleInputGate#requestPartitions(). However, in the current design, this must be under the requestLock which is costly and unnecessary.

I propose a redesign to change that and make the SingleInputGate#requestPartitions() a one-time only thing that should be called before going into any loop asking for new records. In that case, we could keep the lock and ignore any performance-related penalties.

Alternatively, if we had a separate unrequestedPartitions list aside to inputChannels and only go through these when executing SingleInputGate#requestPartitions(), we would change updateInputChannel to always request the newly created partition and will not have the concurrency issue between these two methods anymore. This unrequestedPartitions list would basically also make requestedPartitionsFlag obsolete if we, for example, set it to Optional.empty after finishing the requests.

Issue Links

• duplicates: FLINK-13013 Make sure that SingleInputGate can always request partitions
• supercedes: FLINK-10727 Remove unnecessary synchronization in SingleInputGate#requestPartitions()