[FLINK-36556] Allow to configure starting buffer size when using buffer debloating

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.19.1
• Fix Version/s: 2.0.0
• Component/s: Runtime / Network, Runtime / Task
• Labels: pull-request-available

Release Note

When using buffer debloating, during job's startup or recovery, when we don't know what's the actual records throughput, initial buffer size is now by default 1024B. Previously it was 32 KB, which could lead to jobs choking with large buffers during the startup that took hours to fully process. The starting buffer size can be changed using `taskmanager.memory.starting-segment-size`

• Assignee: Unassigned
• Reporter: Piotr Nowojski
• Created: 16/Oct/24 14:48
• Updated: 25/Oct/24 14:35
• Resolved: 25/Oct/24 14:32

Description

Starting buffer size is 32KB, so during recovery/startup, before backpressure kicks in, Flink job can be flooded with large buffers, completely stalling the progress.

Proposed solution is to make this starting size configurable.

Issue Links

• is related to: FLINK-36555 Guarantee debloated buffer size grows even with very small alpha
• links to: GitHub Pull Request #25574