[FLINK-32774] Reconciliation for autoscaling overrides gets stuck after cancel-with-savepoint

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: kubernetes-operator-1.6.0
• Fix Version/s: kubernetes-operator-1.6.0, kubernetes-operator-1.7.0
• Component/s: Autoscaler, Kubernetes Operator
• Labels: pull-request-available
• Assignee: Gyula Fora
• Reporter: Maximilian Michels
• Created: 07/Aug/23 17:35
• Updated: 09/Aug/23 15:33
• Resolved: 09/Aug/23 15:33

Description

Since https://issues.apache.org/jira/browse/FLINK-32589 the operator does not rely on the Flink configuration anymore to store the parallelism overrides. Instead, it stores them internally in the autoscaler config map. Upon scalings without the rescaling API, the spec is changed on the fly during reconciliation and the parallelism overrides are added.

Unfortunately, this yields to the cluster getting stuck with the job in FINISHED state after taking a savepoint for upgrade. The operator assumes that the new cluster got deployed successfully and goes into DEPLOYED state again.

Log flow (from oldest to newest):

1. Rescheduling new reconciliation immediately to execute scaling operation.
2. Upgrading/Restarting running job, suspending first...
3. Job is in running state, ready for upgrade with SAVEPOINT
4. Suspending existing deployment.
5. Suspending job with savepoint.
6. Job successfully suspended with savepoint
7. The resource is being upgraded
8. Pending upgrade is already deployed, updating status.
9. Observing JobManager deployment. Previous status: DEPLOYING
10. JobManager deployment port is ready, waiting for the Flink REST API...
11. DEPLOYED The resource is deployed/submitted to Kubernetes, but it’s not yet considered to be stable and might be rolled back in the future

It appears the issue might be in (8): https://github.com/apache/flink-kubernetes-operator/blob/c09671c5c51277c266b8c45d493317d3be1324c0/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractFlinkDeploymentObserver.java#L260 because the generation id hasn't been changed by the mere parallelism override change.

Issue Links

• is caused by: FLINK-32589 Carry over parallelism overrides to prevent users from clearing them on updates
• links to: GitHub Pull Request #647
• links to: GitHub Pull Request #649