[FLINK-30652] Use max busytime instead of average to compute true processing rate

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: kubernetes-operator-1.4.0
• Fix Version/s: kubernetes-operator-1.4.0
• Component/s: Autoscaler, Kubernetes Operator
• Labels: pull-request-available
• Assignee: Gyula Fora
• Reporter: Gyula Fora
• Created: 12/Jan/23 13:53
• Updated: 13/Jan/23 20:35
• Resolved: 13/Jan/23 20:35

Description

Currently we use the some of busyTimes and processed records to estimate the true processing rate.

This computation however is not correct when any data skew is present as TPR is not fully additive. The first task to reach 100% utilization will set a limit to the pipeline processing rate through backpressure.

To avoid this we should use the max busyTime and compute the TPR from that.

Issue Links

• links to: GitHub Pull Request #505