Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-12069

Add proper lifecycle management for intermediate result partitions

    XMLWordPrintableJSON

    Details

      Description

      In order to properly execute batch jobs, we should make the lifecycle management of intermediate result partitions the responsibility of the JobMaster/Scheduler component. The Scheduler knows best when an intermediate result partition is no longer needed and, thus, can be freed. So for example, a blocking intermediate result should only be released after all subsequent blocking intermediate results have been completed in order to speed up potential failovers.

      Moreover, having explicit control over intermediate result partitions, could also enable use cases like result partition sharing between jobs and even across clusters (by simply not releasing the result partitions).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chesnay Chesnay Schepler
                Reporter:
                trohrmann Till Rohrmann
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h