Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7897

Invalid use of epoch cache with old message format versions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.1, 2.1.0
    • Fix Version/s: 1.1.2, 2.2.0, 2.1.1, 2.0.2
    • Component/s: None
    • Labels:
      None

      Description

      Message format downgrades are not supported, but they generally work as long as broker/clients at least can continue to parse both message formats. After a downgrade, the truncation logic should revert to using the high watermark, but currently we use the existence of any cached epoch as the sole prerequisite in order to leverage OffsetsForLeaderEpoch. This has the effect of causing a massive truncation after startup which causes re-replication.

      I think our options to fix this are to either 1) clear the cache when we notice a downgrade, or 2) forbid downgrades and raise an error.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                hachikuji Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: