[FLINK-16760] Support the yaml file submission for native Kubernetes integration

• Type: New Feature
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Deployment / Kubernetes, Deployment / Scripts
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Yang Wang
• Created: 25/Mar/20 02:27
• Updated: 17/Nov/21 10:43

Description

Currently, the native K8s integration is friendly and convenient to the Flink users, especially they have some experience of YARN deployment. The submission command and process are very similar and could be integrated into their existing deployer(i.e. job lifecycle management system).

However, if you are a K8s user and prefer the K8s way to start the Flink cluster(regarding it as a application). Then yaml way is more appropriate.

 

What's the difference between standalone on K8s[1] and this ticket?

1. Dynamic resource allocation
2. Do not need to create taskmanager deployment yaml. Since the taskmanagers will be allocated by Flink ResourceManager dynamically on demand. The configmap, jobmanager deployment, service still need to be created manually.
3. Service account needs to be prepared beforehand[2].
4. Some scripts(flink-console.sh, jobmanager.sh, etc.) needs to be updated to use native K8s entrypoint org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint

 

Do we have the alternative option?

A possible way is using a K8s job(yaml file) to run the kubernetes-session.sh to start the Flink cluster. This is just moving the submission from local machine to K8s cluster.

 

[1]. https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html

[2]. https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html#rbac