Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-24892

FLIP-187: Adaptive Batch Scheduler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Implemented
    • None
    • 1.15.0
    • Runtime / Coordination
    • None
    • Hide
      In 1.15, we introduced a new scheduler to Flink: Adaptive Batch Scheduler. The new scheduler can automatically decide parallelisms of job vertices for batch jobs, according to the size of data volume each vertex needs to process.

      More information about this feature and how to enable it could be found in https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
      Show
      In 1.15, we introduced a new scheduler to Flink: Adaptive Batch Scheduler. The new scheduler can automatically decide parallelisms of job vertices for batch jobs, according to the size of data volume each vertex needs to process. More information about this feature and how to enable it could be found in https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-batch-scheduler

    Description

      Introduce a new scheduler to Flink: adaptive batch scheduler. The new scheduler can automatically decide parallelisms of job vertices for batch jobs, according to the size of data volume each vertex needs to process.

      More details see FLIP-187.

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26168 The judgment of chainable ignores StreamExchangeMode when partitioner is ForwardForConsecutiveHashPartitioner

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-26548 the source parallelism is not set correctly with AdaptiveBatchScheduler

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-26692 migrate TpcdsTestProgram.java to new source

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-29663 Further improvements of adaptive batch scheduler

          • Major - Major loss of function.
          • Closed
          1.
          		 Collect sizes of finished BLOCKING result partitions Sub-task Closed Lijie Wang
          2.
          		 Introduce VertexParallelismDecider Sub-task Closed Lijie Wang
          3.
          		 Job finishes iff all job vertices finish Sub-task Closed Lijie Wang
          4.
          		 Allow to create execution vertices and execution edges lazily Sub-task Closed Lijie Wang
          5.
          		 Support flexible number of subpartitions in IntermediateResultPartition Sub-task Closed Lijie Wang
          6.
          		 Let some scheduler components updatable Sub-task Closed Lijie Wang
          7.
          		 Support calcuate network memory for dynamic graph. Sub-task Closed Lijie Wang
          8.
          		 Support register operator coordinators for newly initialized ExecutionJobVertex. Sub-task Closed Lijie Wang
          9.
          		 Shuffle Service Supports Consuming Subpartition Range Sub-task Closed Lijie Wang
          10.
          		 Introduce vertex wise scheduling strategy Sub-task Closed Lijie Wang
          11.
          		 Introduce AdaptiveBatchScheduler Sub-task Closed Lijie Wang
          12.
          		 Introduce job property isDynamicGraph to ExecutionConfig Sub-task Closed Lijie Wang
          13.
          		 Introduce ForwardForConsecutiveHashPartitioner Sub-task Closed Lijie Wang
          14.
          		 Make implicit assumption of SQL local hash explicit Sub-task Closed godfrey he
          15.
          		 Convert unspecified edge to rescale when using adaptive batch scheduler Sub-task Closed Lijie Wang
          16.
          		 Change the format of adaptive batch scheduler config option to "jobmanager.adaptive-batch-scheduler.XXX" Sub-task Closed Lijie Wang
          17.
          		 Add e2e TPCDS tests to run against the AdaptiveBatchScheduler Sub-task Closed Lijie Wang
          18.
          		 Add documentation about the AdaptiveBatchScheduler Sub-task Closed Lijie Wang
          19.
          		 Test Adaptive Batch Scheduler manually Sub-task Closed Niklas Semmler

          Activity

            People

              wanglijie Lijie Wang
              wanglijie Lijie Wang
              Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: