[FLINK-9953] Active Kubernetes integration

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: Deployment / Kubernetes, Runtime / Coordination
• Labels: pull-request-available
• Assignee: Yang Wang
• Reporter: Till Rohrmann
• Created: 25/Jul/18 15:22
• Updated: 17/May/22 09:04
• Resolved: 08/Jan/20 11:15

Description

This is the umbrella issue tracking Flink's active Kubernetes integration. Active means in this context that the ResourceManager can talk to Kubernetes to launch new pods similar to Flink's Yarn and Mesos integration.

Phase1 implementation will have complete functions to make flink running on kubernetes. Only session cluster is supported.

Phrase2 is mainly focused on production optimization, including per-job cluster, k8s native high-availability, storage, network, log collector and etc.

Issue Links

• is blocked by: FLINK-10939 Add documents for natively running Flink session cluster on k8s
• is related to: FLINK-16111 Kubernetes deployment does not respect "taskmanager.cpu.cores".
• is related to: FLINK-17709 Active Kubernetes integration phase 3 - Advanced Features
• relates to: FLINK-14460 Active Kubernetes integration phase 2 - Advanced Features
• relates to: FLINK-15788 Various Kubernetes integration improvements
• links to: GitHub Pull Request #8856
• links to: Kubernetes native integration Implementation

Sub-Tasks

• Initial flink-kubernetes module with empty implementation: Closed
• Kubernetes ClusterDescriptor: Closed
• Implement ResourceManager for Kubernetes: Closed
• Implement KubernetesSessionClusterEntrypoint: Closed
• Implement TaskManager Entrypoint: Closed
• Implement KubeClient with Faric8 Kubernetes clients: Closed
• Implement Command line tools: Closed
• Add entrypoint scripts for k8s: Closed
• Add e2e test for natively running Flink session cluster on Kubernetes: Resolved
• Service selector needs to contain jobmanager component label: Closed
• Kubernetes should not have a CustomCommandLine.: Closed
• Remove unused KubernetesConfigOptionsInternal.ENTRY_POINT_CLASS_ARGS: Closed
• Add resource related config options to dynamical properties for Kubernetes: Resolved
• Uniformize Kubernetes executor name with the rest: Closed
• Fabric8FlinkKubeClient doesn't work if ingress has hostname but no IP: Closed
• Starting jobmanager pod should respect containerized.heap-cutoff: Closed