[FLINK-31976] Once marked as an inefficient scale-up, further scaling may not happen forever

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Fix Version/s: kubernetes-operator-1.5.0
• Component/s: Autoscaler
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Tan Kim
• Created: 30/Apr/23 08:37
• Updated: 06/May/23 12:02
• Resolved: 03/May/23 15:58

Description

The determination of whether it is an inefficient scale-up is calculated as follows

double lastProcRate = lastSummary.getMetrics().get(TRUE_PROCESSING_RATE).getAverage();
double lastExpectedProcRate =
lastSummary.getMetrics().get(EXPECTED_PROCESSING_RATE).getCurrent();
var currentProcRate = evaluatedMetrics.get(TRUE_PROCESSING_RATE).getAverage();
double expectedIncrease = lastExpectedProcRate - lastProcRate;
double actualIncrease = currentProcRate - lastProcRate;

boolean withinEffectiveThreshold =
(actualIncrease / expectedIncrease)
>= conf.get(AutoScalerOptions.SCALING_EFFECTIVENESS_THRESHOLD);

Because the expectedIncrease value references the last scaling history, it will not change unless there is an additional scale-up, only the actualIncrease value will change.
The actualIncrease value is currentProcRate( avg of TRUE_PROCESSING_RATE),
The calculation of TRUE_PROCESSING_RATE is as follows
trueProcessingRate = busyTimeMultiplier * numRecordsInPerSecond.getSum()

For example, let's say you've been marked as an inefficient scale-up, but the LAG continues to build up.
You need to scale up to eliminate the growing LAG, but because you're marked as an inefficient scale-up, it won't happen.
To unmark a scaleup as inefficient, the following conditions must be met: actualIncrease/expectedIncrease > SCALING_EFFECTIVENESS_THRESHOLD (default 0.1)

Here, expectedIncrease is a constant with lastSummary, so the value of actualIncrease must increase.
However, the actualIncrease value is proportional to busyTimeMultiplier and numRecordsInPerSecond, and these two values will converge to a certain value if no scaling occurs.
Therefore, the value of actualIncrease will also converge.
If this value fails to cross a threshold, no further scaling up is possible, even if the lag continues to build up.

Issue Links

• links to: GitHub Pull Request #579