Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5241

GlobalKTable does not checkpoint offsets after restoring state

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10.2.1
    • Fix Version/s: 0.10.2.2, 0.11.0.0
    • Component/s: streams
    • Labels:
      None

      Description

      I'm experimenting with an application that uses a relatively large GlobalKTable, and noticed that streams was not checkpointing its offsets on close(). This is because although org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl#restoreState updates the checkpoint map, the actual checkpointing itself is guarded by a check that the offsets passed from the GloablStateUpdateTask are not empty. This is frustrating because if the topic backing the global table is both large (therefore taking a long time to restore) and infrequently written, then streams rebuilds the table from scratch every time the application is started.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                twbecker Tommy Becker
                Reporter:
                twbecker Tommy Becker
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: