[FLINK-19206] Add an ability to set ownerReference manually in Kubernetes

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Resolved
• Component/s: Deployment / Kubernetes
• Assignee: Unassigned
• Reporter: Mike Kaplinskiy
• Created: 12/Sep/20 06:08
• Updated: 17/Mar/21 02:35
• Resolved: 17/Mar/21 02:35

Description

The current Kubernetes deployment creates a service that is the ownerReference of all the sub-objects (the JM & TM deployments & the rest service). However, something presumably has to start the cluster in the first place. If you are using a job cluster, that can be something like a kubernetes Job, a CronJob or a tool like Airflow. Unfortunately any failures in the Flink job can cause retries from these higher-level primitives, which can yield a lot of "stale clusters" that aren't GCed.

The proposal here is to add a configuration option to set the ownerReference of the Flink Service. This way the service (and by proxy, all the cluster components) get deleted when the "parent" decides - including if the parent is itself a Kubernetes pod. For reference, Spark does something similar via spark.kubernetes.driver.pod.name (documented at https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode-executor-pod-garbage-collection).

Issue Links

• is superceded by: FLINK-20359 Support adding Owner Reference to Job Manager in native kubernetes setup