Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5307

Concurrent offset commit on same partition may result in inconsistent cache

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: offset manager
    • Labels:
      None

      Description

      Consider two concurrent offset commits: the first one at offset A, the second at offset B. It is possible for both offsets to be appended to the log before either of them had been successfully acknowledged. Currently we maintain in the group metadata a collection of pending offset commits, but we only store one offset for each partition. When the commit for A is received, we add an entry for that partition with A. When the commit for B arrives, we would overwrite that value with B. Now two cases where this results in incorrect behavior.

      1. After the callback for A is invoked, we materialize whatever offset is in the pending offset collection unconditionally. In this case, we would incorrectly materialize B even though it had not been successfully written. Later B may fail and we have inconsistent state.
      2. It could happen that the callback for B is invoked before A. If the result was successful, no harm done, but if it failed, then we need to restore A since it may still be completed successfully. If it does complete successfully, then we'll have inconsistent state again.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hachikuji Jason Gustafson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: