Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8549

Buffered state during checkpointing may be keyed incorrectly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P1
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.17.0
    • Component/s: runner-flink
    • Labels:
      None

      Description

      The current buffer logic for items emitted during checkpointing is faulty in the sense that the buffer is partitioned on the output keys of the operator. The key may be changed or even be dropped. Thus, the original key partitioning will not be maintained which will cause checkpointing to fail.

      An alternative solution would be BEAM-6733, but we may want to keep the current buffering logic in place. The proposed solution here is to remove the keyed buffering. The output buffer may then always be redistributed round-robin upon restoring from a checkpoint. Note that this is fine because no assumption can be made about the distribution of output elements of an operation.

        Attachments

          Activity

            People

            • Assignee:
              mxm Maximilian Michels
              Reporter:
              mxm Maximilian Michels
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 10m
                1h 10m