[FLINK-30574] Do not scale further up if last scaling was ineffective

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: kubernetes-operator-1.4.0
• Component/s: Autoscaler, Kubernetes Operator
• Labels: pull-request-available
• Assignee: Gyula Fora
• Reporter: Gyula Fora
• Created: 05/Jan/23 10:00
• Updated: 12/Jan/23 13:51
• Resolved: 12/Jan/23 13:51

Description

Related to https://issues.apache.org/jira/browse/FLINK-30571 but somewhat simpler. 

It should be easy to detect when a scale up operation did not significantly increase the parallelism (or even deteriorated it). In these cases it is very important to not try to scale further.

This can happen if the job hit some external limit (such as rate limiting on an external service) or if the performance problems are caused by other factors such as memory.

Issue Links

• links to: GitHub Pull Request #502