[FLINK-32774] Reconciliation for autoscaling overrides gets stuck after cancel-with-savepoint - ASF JIRA

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: kubernetes-operator-1.6.0
Fix Version/s: kubernetes-operator-1.6.0, kubernetes-operator-1.7.0
Component/s: Autoscaler, Kubernetes Operator
Labels:
- pull-request-available

Description

Since https://issues.apache.org/jira/browse/FLINK-32589 the operator does not rely on the Flink configuration anymore to store the parallelism overrides. Instead, it stores them internally in the autoscaler config map. Upon scalings without the rescaling API, the spec is changed on the fly during reconciliation and the parallelism overrides are added.

Unfortunately, this yields to the cluster getting stuck with the job in FINISHED state after taking a savepoint for upgrade. The operator assumes that the new cluster got deployed successfully and goes into DEPLOYED state again.

Log flow (from oldest to newest):

Rescheduling new reconciliation immediately to execute scaling operation.
Upgrading/Restarting running job, suspending first...
Job is in running state, ready for upgrade with SAVEPOINT
Suspending existing deployment.
Suspending job with savepoint.
Job successfully suspended with savepoint
The resource is being upgraded
Pending upgrade is already deployed, updating status.
Observing JobManager deployment. Previous status: DEPLOYING
JobManager deployment port is ready, waiting for the Flink REST API...
DEPLOYED The resource is deployed/submitted to Kubernetes, but it’s not yet considered to be stable and might be rolled back in the future