[FLINK-9142] Lower the minimum number of buffers for incoming channels to 1

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.0, 1.6.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: boshu Zheng
• Reporter: Nico Kruber
• Created: 05/Apr/18 13:32
• Updated: 18/Mar/20 06:55
• Resolved: 18/Mar/20 06:55

Description

Even if we make the floating buffers optional, we still require taskmanager.network.memory.buffers-per-channel number of (exclusive) buffers per incoming channel with credit-based flow control while without, the minimum was 1 and only the maximum number of buffers was influenced by this parameter.

taskmanager.network.memory.buffers-per-channel is set to 2 by default with the argumentation that this way we will have one buffer available for netty to process while a worker thread is processing/deserializing the other buffer. While this seems reasonable, it does increase our minimum requirements. Instead, we could probably live with 1 exclusive buffer and up to gate.getNumberOfInputChannels() * (networkBuffersPerChannel - 1) + extraNetworkBuffersPerGate floating buffers. That way we will have the same memory footprint as before with only slightly changed behaviour.

Issue Links

• is superceded by: FLINK-16012 Reduce the default number of exclusive buffers from 2 to 1 on receiver side
• links to: GitHub Pull Request #7579