Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-12069

Add proper lifecycle management for intermediate result partitions

    XMLWordPrintableJSON

Details

    Description

      In order to properly execute batch jobs, we should make the lifecycle management of intermediate result partitions the responsibility of the JobMaster/Scheduler component. The Scheduler knows best when an intermediate result partition is no longer needed and, thus, can be freed. So for example, a blocking intermediate result should only be released after all subsequent blocking intermediate results have been completed in order to speed up potential failovers.

      Moreover, having explicit control over intermediate result partitions, could also enable use cases like result partition sharing between jobs and even across clusters (by simply not releasing the result partitions).

      Attachments

        Issue Links

          Activity

            People

              chesnay Chesnay Schepler
              trohrmann Till Rohrmann
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h