[FLINK-23452] Measuring subtask throughput

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Labels: pull-request-available
• Assignee: Anton Kalashnikov
• Reporter: Anton Kalashnikov
• Created: 21/Jul/21 12:01
• Updated: 23/Sep/21 18:10
• Resolved: 29/Jul/21 12:02

Description

In the first implementation, throughput could be measured for the whole subtask. The throughput calculation should take into account the numbers of bytes that were handled, the backpressure time and ignore the idle time. The main idea is to keep the balance between idle and backpressure time, so if the backpressure time is high we should decrease the buffer size to provide the configured handling time and vice versa if the subtask is idle time that period should be ignored from calculating the throughput. Otherwise, in the case of network bottleneck, we might have ended up with a small buffer size that’s causing the bottleneck in the first place but we are not able to increase it due to idle time reducing throughput and lowering the buffer size.

Issue Links

• causes: FLINK-23558 E2e tests fail because of quiesced system timers service
• causes: FLINK-23560 Performance regression on 29.07.2021
• causes: FLINK-23590 StreamTaskTest#testProcessWithUnAvailableInput is flaky
• is related to: FLINK-23680 StreamTaskTimerTest.testOpenCloseAndTimestamps fails on azure
• links to: GitHub Pull Request #16556