Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-13371

Release partitions in JM if producer restarts

    XMLWordPrintableJSON

    Details

      Description

      As discussed in FLINK-13245, there can be a case that producer does not even detect any consumption attempt if consumer fails before the connection is established. It means we cannot fully rely on shuffle service for the release on consumption in case of consumer failure. When producer restarts it will leak partitions from the previous attempt. Previously we had an explicit release call for this case in Execution.cancel/suspend. Basically JM has to explicitly release all partitions produced by the previous task execution attempt in case of producer restart, including `released on consumption` partitions. For this change, we might need to track all partitions in PartitionTrackerImpl.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chesnay Chesnay Schepler
                Reporter:
                azagrebin Andrey Zagrebin
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m