Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-16430

FLIP-119 Pipelined Region Scheduling

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Done
    • 1.11.0
    • 1.12.0
    • Runtime / Coordination
    • None
    • Hide
      Beginning from Flink 1.12, jobs will be scheduled in the unit of pipelined regions. A pipelined region is a set of pipelined connected tasks. This means that, for streaming jobs which consist of multiple regions, it no longer waits for all tasks to acquire slots before starting to deploy tasks. Instead, any region can be deployed once it has acquired enough slots for within tasks. For batch jobs, tasks will not be assigned slots and get deployed individually. Instead, a task will be deployed together with all other tasks in the same region, once the region has acquired enough slots.
      Show
      Beginning from Flink 1.12, jobs will be scheduled in the unit of pipelined regions. A pipelined region is a set of pipelined connected tasks. This means that, for streaming jobs which consist of multiple regions, it no longer waits for all tasks to acquire slots before starting to deploy tasks. Instead, any region can be deployed once it has acquired enough slots for within tasks. For batch jobs, tasks will not be assigned slots and get deployed individually. Instead, a task will be deployed together with all other tasks in the same region, once the region has acquired enough slots.

    Description

      Pipelined region scheduling is targeting to allow batch jobs with PIPELINED data exchanges to run without the risk to encounter a resource deadlock.

      More details see FLIP-119

      Attachments

        Issue Links

        is a parent of

        New Feature - A new feature of the product, which has yet to be developed. FLINK-18689 Deterministic Slot Sharing

        • Major - Major loss of function.
        • Closed
        is blocked by

        Sub-task - The sub-task of the issue FLINK-14162 Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies

        • Major - Major loss of function.
        • Closed
        is related to

        Bug - A problem which impairs or prevents the functions of the product. FLINK-19142 Local recovery can be broken if slot hijacking happened during a full restart

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. FLINK-19923 Remove unused BulkSlotProvider, its implementation and tests

        • Major - Major loss of function.
        • Closed
        relates to

        New Feature - A new feature of the product, which has yet to be developed. FLINK-10643 Bubble execution: Resource aware job execution

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        links to

        Web Link FLIP-119

        mentioned in

        Page Loading...

        Page Loading...

        1.
        		 Replace FailoverTopology with SchedulingTopology Sub-task Closed Gary Yao Actions
        2.
        		 Add PipelinedRegion Interface to Topology Sub-task Closed Gary Yao Actions
        3.
        		 Implement PipelinedRegionSchedulingStrategy Sub-task Closed Zhu Zhu Actions
        4.
        		 Introduce GlobalDataExchangeMode for JobGraph Generation Sub-task Closed Zhu Zhu Actions
        5.
        		 Blink Planner set GlobalDataExchangeMode Sub-task Closed Zhu Zhu   Actions
        6.
        		 Simplify SchedulingStrategy#onPartitionConsumable(...) parameters Sub-task Closed Zhu Zhu Actions
        7.
        		 Implement PipelinedRegion interface for SchedulingTopology Sub-task Closed Gary Yao   Actions
        8.
        		 Drop generic Types in SchedulingTopology Interface Sub-task Closed Gary Yao Actions
        9.
        		 Migrate RestartPipelinedRegionFailoverStrategyBuildingTest to PipelinedRegionComputeUtilTest Sub-task Closed Gary Yao   Actions
        10.
        		 Implements bulk allocation for physical slots Sub-task Closed Zhu Zhu   Actions
        11.
        		 Introduce PreferredLocationsRetriever Sub-task Closed Zhu Zhu   Actions
        12.
        		 Allocates slots in bulks on pipelined region scheduling Sub-task Closed Zhu Zhu   Actions
        13.
        		 Implement FIFO Physical Slot Assignment in SlotPoolImpl Sub-task Closed Zhu Zhu   Actions
        14.
        		 Integrate pipelined region scheduling Sub-task Closed Zhu Zhu   Actions
        15.
        		 Unify slot request timeout handling for streaming and batch tasks Sub-task Closed Zhu Zhu   Actions
        16.
        		 Avoid scheduling deadlocks caused by cyclic input dependencies between regions Sub-task Closed Zhu Zhu   Actions
        17.
        		 Expose more details in logs for debugging bulk slot allocation failures Sub-task Closed Unassigned   Actions
        18.
        		 Simplify tests of SlotPoolImpl Sub-task Closed Zhilong Hong Actions
        19.
        		 Enable pipelined scheduling by default Sub-task Closed Zhu Zhu   Actions
        20.
        		 Improve pipelined region scheduling performance Sub-task Closed Zhu Zhu   Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zhuzh Zhu Zhu
            zhuzh Zhu Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 337h 40m
                337h 40m

                Slack

                  Issue deployment