[FLINK-23974] Decreased throughput with enabled buffer debloat

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.14.0
• Fix Version/s: 1.15.0
• Assignee: Anton Kalashnikov
• Reporter: Anton Kalashnikov
• Created: 25/Aug/21 16:57
• Updated: 24/Jan/22 14:58
• Resolved: 24/Jan/22 14:58

Description

According to task https://issues.apache.org/jira/browse/FLINK-23456, we have some performance drop when the buffer debloat is enabled:

• for static load with throttling in source - 1-3%
• for sinus shape load with throttling in source - 2-3%
• for erratic load with throttling in source - 5-6%

It needs to investigate the reason for that and try to improve if it is possible.

It makes sense to write the microbenchmarks for these scenarios to reproduce the problem locally.

Problem assumption:
Highly likely the problem is the speed of reaction on increasing load. During the investigation, it needs to pay attention to the calculation throughput for different load profiles. It is possible that when the load is low we are decreasing the buffer size to a small value(maybe minimum value) and when the load gets back to high we need several iterations of the throughput calculation to reach the appropriate buffer size.

Issue Links

• Dependent: FLINK-24578 Unexpected erratic load shape for channel skew load profile and ~10% performance loss with enabled debloating
• Dependent: FLINK-24671 Possible NPE in LocalInputChannel#getBuffersInUseCount before initialization of subpartitionView
• Dependent: FLINK-24467 Set min and max buffer size even if the difference less than threshold
• Dependent: FLINK-24468 NPE when notifyNewBufferSize
• Dependent: FLINK-24470 EMA for buffer debloat level
• Dependent: FLINK-24553 Change buffer debloating default configuration values