[FLINK-20219] Rethink the HA related ZNodes/ConfigMap clean up for session cluster

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.12.0
• Component/s: Deployment / Kubernetes, Deployment / Scripts, Runtime / Coordination
• Labels: pull-request-available stale-assigned
• Assignee: Yang Wang
• Reporter: Yang Wang
• Created: 18/Nov/20 12:07
• Updated: 23/Feb/23 08:56

Description

When I am testing the Kubernetes HA service, I realize that ConfigMap clean up for session cluster(both standalone and native) are not very easy.

• For the native K8s session, we suggest our users to stop it via echo 'stop' | ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=<ClusterID> -Dexecution.attached=true. Currently, it has the same effect with kubectl delete deploy <ClusterID>. This will not clean up the leader ConfigMaps(e.g. ResourceManager, Dispatcher, RestServer, JobManager). Even though there is no running jobs before stop, we still get some retained ConfigMaps. So when and how to clean up the retained ConfigMaps? Should the user do it manually? Or we could provide some utilities in Flink client.
• For the standalone session, I think it is reasonable for the users to do the HA ConfigMap clean up manually.

 

We could use the following command to do the manually clean up.

kubectl delete cm --selector='app=<ClusterID>,configmap-type=high-availability'

 

Note: This is not a problem for Flink application cluster. Since we could do the clean up automatically when all the running jobs in the application reached terminal state(e.g. FAILED, CANCELED, FINISHED) and then destroy the Flink cluster.

Issue Links

• duplicates: FLINK-30813 Residual zk data when using the kubernetes session mode
• relates to: FLINK-26750 HA cluster cleanup not complete
• links to: GitHub Pull Request #16121
• mentioned in: Page Loading...