Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9196

Records exposed before advancement of high watermark after segment roll

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0
    • Component/s: None
    • Labels:
      None

      Description

      We cache the segment position of the high watermark and last stable offset inside `Log`. There is no logic currently to update the cached position when the segment rolls.

      Suppose we have a log with one segment (0.log). We write 5 records and update the high watermark to match the log end offset. The cached segment position will be something like LogOffsetMetadata(offset=5, segment=0, position=100). Now suppose we roll to segment 5.log and write some new data. If a consumer fetches from offset 5, then the current fetch logic will find segment 5.log and incorrectly use position 100 in this segment. The result is that data from the new segment gets exposed prematurely.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                hachikuji Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: