Details

    Description

      Introduce a pluginable decorators mechanism into Flink JobManager and TaskManager. Currently, all decorators are hard-coded in the JobManager deployment and TaskManager pod K8S creation.

       

      We propose using SPI or Flink plugins mechanism to load the external decorators. In this way, we can make Flink more flexible towards supporting the decorators of customized K8S schedulers.

      We propose a new plugin mechanism for supporting load the jar package of a single customized scheduler. The driver jar package should contain the specific K8sStepDecorator and its dependencies. Taking an example, the jar package should be packaged into the ‘opt’ directory, and contains 2 major things:

      1. A K8sStepDecorator implemented by Customized K8S SchedulerA.
      2. The all dependencies from the introduced K8sStepDecorator.

      when users want to use the customized scheduler A, he/she need to create a new directory which named as the customized scheduler’s name A, then copy & paste the said jar into the new directory, that could be loaded when user specific the related Flink K8S configuration options and enable the functionality in Flink.

      Attachments

        Issue Links

          Activity

            People

              bzhaoop bo zhao
              bzhaoop bo zhao
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: