Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6854

Log cleaner fails with transaction markers that are deleted during clean

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.0.2, 1.1.1, 2.0.0
    • Component/s: core
    • Labels:
      None

      Description

      Log cleaner grows buffers when `result.messagesRead` is zero. In a typical scenario, this is a result of source buffer being too small to read the first batch. The buffer is then doubled in size until one batch can be read, up to a maximum of `max.message.size`. There are issues with the maximum message size used in calculations as reported in KAFKA-6834. But there is a separate issue with the use of `result.messagesRead` when transactions are used. This contains the number of filtered messages read from source which can be zero when a transaction control marker is discarded. Log cleaner incorrectly assumes that messages were not read because the buffer was too small. This can result in the log cleaner attempting to grow buffers to double the buffer size unnecessarily, failing with an exception if the buffer is already `max.message.bytes`. This kills the log cleaner.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rsivaram Rajini Sivaram
                Reporter:
                rsivaram Rajini Sivaram
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: