Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8306

Ensure consistency of checkpointed log start offset and current log end offset

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2, 2.1.2, 2.2.1
    • Component/s: None
    • Labels:
      None

      Description

      When initializing a log, we may use the checkpointed log start offset. We need to ensure that the log end offset is set consistently with this value (i.e. it must be greater than or equal to it). This may not always be true if the log data is removed or has become corrupted. As a simple experiment, you can try the following steps to reproduce the problem:

      1. Write some data to the partition
      2. Use DeleteRecords to advance log start
      3. Shutdown the broker
      4. Delete the log directory
      5. Restart the broker

      You will see something like this in the logs:

      [2019-04-29 11:55:21,259] INFO [Log partition=foo-0, dir=/tmp/kafka-logs] Completed load of log with 1 segments, log start offset 10 and log end offset 0 in 36 ms (kafka.log.Log)

      This may be the cause of KAFKA-8255, but I am not sure yet.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dhruvilshah Dhruvil Shah
                Reporter:
                hachikuji Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: