Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11298

Scheduling job in the unit of concurrent groups

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Abandoned
    • None
    • None
    • Runtime / Coordination
    • None

    Description

      Now flink only support two scheduling modes, that's scheduling all tasks Eager for streaming jobs and scheduling all task Lazy_from_source for batch jobs. This is not flexible enough for the various requirements of different job such as FLINK-10240. We proposal a new ConcurrentSchedulingGroup based scheduling strategy which first split a job into serval concurrent groups and then schedule it in the unit of concurrent groups. This strategy will support not only the existing EAGER and LAZY_FROM_SOURCE mode but also other situation such as the Build/Probe in FLINK-10240.

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue FLINK-10432 Introduce bulk/group-aware scheduling

          • Major - Major loss of function.
          • Closed
          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. FLINK-10429 Redesign Flink Scheduling, introducing dedicated Scheduler component

          • Major - Major loss of function.
          • Closed
          links to

          Web Link design doc

          Activity

            People

              tiemsn shuai.xu
              tiemsn shuai.xu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: