[FLINK-26559] Support Customized Kubernetes Schedulers for Flink Kubernetes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 1.20.0
• Component/s: Deployment / Kubernetes
• Assignee: bo zhao
• Reporter: bo zhao
• Created: 10/Mar/22 01:55
• Updated: 11/Mar/24 12:44

Description

This is an umbrella issue for tracking the work for supporting Volcano on FLink Kubernetes. The schedulers like Volcano provide more YARN like features (such as queues and minimum resources before scheduling jobs) that many folks want on Kubernetes.

 

Volcano is a graduated CNCF project (sig-batch).

 

They've taken slightly different approaches to solving the same problem, but there must have the same high level logic code in Flink.

FLIP : https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal

DISCUSSION: https://lists.apache.org/thread/pf8dvbvqf845wh0x63z68jmhh4pvsbow

VOTE:  https://lists.apache.org/thread/w1tpl5cwdrj7bc8qrlp96q25y9sv5gjk

VOTE RESULT: https://lists.apache.org/thread/hl71o9kvfg1lktnkq4nvos07pjthzkkl

Design DOC: [DRAFT] https://docs.google.com/document/d/1n0ZF4ssM8syLhMy1XAiv9LiU3Ig7udOeWaKi60mXQV8/edit?usp=sharing

 

 

Sub-Tasks

• [1] Support specify K8S pod scheduler: Open
• [2] Need the ability of preparing K8S resource before JobManager creation: Open
• [3] Support pluginable decorators mechanism: Open
• [3.4] Provide a way to exclude the build-in stepDecorator: Open
• [5] Support customized scheduler options as a hashmap for each scheduler.: Open