Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21075

FLIP-160: Adaptive scheduler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.13.0
    • 1.13.0
    • Runtime / Coordination
    • None

    Description

      The adaptive scheduler is a new scheduler which first declares the required resources and waits for them before deciding on the parallelism with which to execute a job. This behaviour has the benefit that Flink will be able to adjust the job to the available resources and to also make use of resources which arrive later. The details can be found in the FLIP-160.

      This ticket serves as the umbrella for all the MVP targeted for Flink 1.13.0 (as described in FLIP-160). Improvements beyond the MVP are captured in there own tickets.

      Attachments

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. FLINK-10407 FLIP-159: Reactive mode

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. FLINK-21510 ExecutionGraph metrics collide on restart

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-21450 Add local recovery support to adaptive scheduler

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-21439 Adaptive Scheduler: Add support for exception history

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-21846 Rethink whether failure of ExecutionGraph creation in Adaptive Scheduler should directly fail the job

          • Not a Priority -
          • Open

          Bug - A problem which impairs or prevents the functions of the product. FLINK-21954 Test failures occur due to the test not waiting for the ExecutionGraph to be created in Adaptive Scheduler

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. FLINK-21964 HA e2e test failed due to not enough resources being available with Adaptive Scheduler

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-22255 AdaptiveScheduler improvements/bugs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Technical Debt - FLINK-21604 Remove adaptive scheduler nightly test profile, replace with more specific testing approach

          • Not a Priority -
          • Open
          links to

          Web Link FLIP-160

          mentioned in

          Page Loading...

          Page Loading...

          1.
          		 Document how to use the declarative scheduler and its limitations Sub-task Closed Robert Metzger
          2.
          		 Abstract SlotPool from JobManager by introducing SlotPoolService Sub-task Closed Till Rohrmann
          3.
          		 Add DeclarativeSlotPoolService Sub-task Closed Till Rohrmann
          4.
          		 Add SlotAllocator Sub-task Closed Chesnay Schepler
          5.
          		 Introduce JobType to distinguish between batch and streaming jobs Sub-task Closed Robert Metzger
          6.
          		 Add DeclarativeScheduler skeleton Sub-task Closed Chesnay Schepler
          7.
          		 Set up cron job to run CI with declarative scheduler Sub-task Closed Robert Metzger
          8.
          		 Add ScaleUpController Sub-task Closed Robert Metzger
          9.
          		 Add DeclarativeScheduler / Created state Sub-task Closed Robert Metzger
          10.
          		 Add DeclarativeScheduler / WaitingForResources state Sub-task Closed Robert Metzger
          11.
          		 Add DeclarativeScheduler / Executing state Sub-task Closed Robert Metzger
          12.
          		 Add DeclarativeScheduler / Restarting state Sub-task Closed Robert Metzger
          13.
          		 Add DeclarativeScheduler / Canceling state Sub-task Closed Robert Metzger
          14.
          		 Add DeclarativeScheduler / Failing state Sub-task Closed Robert Metzger
          15.
          		 Add DeclarativeScheduler / Finished state Sub-task Closed Robert Metzger
          16.
          		 Introduce stopping with savepoint state Sub-task Closed Robert Metzger
          17.
          		 Add tests for StateWithExecutionGraph Sub-task Closed Robert Metzger
          18.
          		 Add resourceTimeout configuration Sub-task Closed Chesnay Schepler
          19.
          		 Remove State#onEnter Sub-task Closed Robert Metzger
          20.
          		 Adjust queryable-state JobMaster tests Sub-task Closed Chesnay Schepler
          21.
          		 Restarting/Canceling/Failing expose globally-terminal EG Sub-task Closed Chesnay Schepler
          22.
          		 Rename DeclarativeScheduler to AdaptiveScheduler Sub-task Closed Till Rohrmann
          23.
          		 Adjust JobMasterTest#testRestoring[...]FromSavepoint Sub-task Closed Till Rohrmann
          24.
          		 Adjust JobMasterTest#testRequestNextInputSplitWith*Failover Sub-task Closed Chesnay Schepler
          25.
          		 Attempt numbers are not maintained across restarts Sub-task Closed Chesnay Schepler
          26.
          		 Tests that manually build a JobGraph cannot use DeclarativeScheduler Sub-task Closed Till Rohrmann
          27.
          		 Consolidate Scheduler/SlotPool factories Sub-task Closed Till Rohrmann
          28.
          		 Some tests expect to fail if parallelism cannot be met Sub-task Closed Chesnay Schepler
          29.
          		 YARNITCase runs indefinitely Sub-task Closed Chesnay Schepler
          30.
          		 SchedulerNGFactoryFactoryTest fails if DeclarativeScheduler is enabled via property Sub-task Closed Unassigned
          31.
          		 Add numRestarts metric Sub-task Closed Chesnay Schepler
          32.
          		 StateWithExecutionGraph.suspend fails with IllegalStateException Sub-task Closed Till Rohrmann
          33.
          		 Creation of ExecutionGraph happens in main thread Sub-task Closed Till Rohrmann
          34.
          		 UnalignedCheckpointITCase does not terminate Sub-task Closed Unassigned
          35.
          		 FileSourceTextLinesITCase.testContinuousTextFileSourceWithJobManagerFailover fails Sub-task Closed Matthias Pohl
          36.
          		 Document Metrics Limitations Sub-task Closed Chesnay Schepler

          Activity

            People

              rmetzger Robert Metzger
              trohrmann Till Rohrmann
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: