[FLINK-33773] Add fairness to scaling decisions

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: kubernetes-operator-1.11.0
• Component/s: Autoscaler, Deployment / Kubernetes
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 07/Dec/23 14:50
• Updated: 14/Oct/24 16:53

Description

The current scaling logic is inherently unfair. In a scenario of heavy backlog, whichever pipelines come first, they will end up taking most of the resources. Some kind of fairness should be introduced, for example:

• Cap the max number of resulting pods at a % of the cluster resources
• Allow scale up round-robin across all pipelines