Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4019

LogCleaner should grow read/write buffer to max message size for the topic

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: 0.10.1.0
    • Component/s: core
    • Labels:
      None

      Description

      Currently, the LogCleaner.growBuffers() only grows the buffer up to the default max message size. However, since the max message size can be customized at the topic level, LogCleaner should allow the buffer to grow up to the max message allowed by the topic. Otherwise, the cleaner will get stuck on a large message.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/1758

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1758
          Hide
          hachikuji Jason Gustafson added a comment -

          Issue resolved by pull request 1758
          https://github.com/apache/kafka/pull/1758

          Show
          hachikuji Jason Gustafson added a comment - Issue resolved by pull request 1758 https://github.com/apache/kafka/pull/1758
          Hide
          ijuma Ismael Juma added a comment -

          We probably want this for 0.10.1.0.

          Show
          ijuma Ismael Juma added a comment - We probably want this for 0.10.1.0.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user rajinisivaram opened a pull request:

          https://github.com/apache/kafka/pull/1758

          KAFKA-4019: Update log cleaner to handle max message size of topics

          Grow read and write buffers of cleaner up to the maximum message size of the log being cleaned if the topic has larger max message size than the default config of the broker.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/rajinisivaram/kafka KAFKA-4019

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1758.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1758


          commit fa89ba94cbc2abc54a94497c3207bb2bce62743a
          Author: Rajini Sivaram <rajinisivaram@googlemail.com>
          Date: 2016-08-18T08:08:17Z

          KAFKA-4019: Update log cleaner to handle max message size of topics


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user rajinisivaram opened a pull request: https://github.com/apache/kafka/pull/1758 KAFKA-4019 : Update log cleaner to handle max message size of topics Grow read and write buffers of cleaner up to the maximum message size of the log being cleaned if the topic has larger max message size than the default config of the broker. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rajinisivaram/kafka KAFKA-4019 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1758.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1758 commit fa89ba94cbc2abc54a94497c3207bb2bce62743a Author: Rajini Sivaram <rajinisivaram@googlemail.com> Date: 2016-08-18T08:08:17Z KAFKA-4019 : Update log cleaner to handle max message size of topics

            People

            • Assignee:
              rsivaram Rajini Sivaram
              Reporter:
              junrao Jun Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development