Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3894

LogCleaner thread crashes if not even one segment can fit in the offset map

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.2, 0.9.0.1, 0.10.0.0
    • Fix Version/s: 0.10.1.0
    • Component/s: core
    • Labels:
    • Environment:
      Oracle JDK 8
      Ubuntu Precise

      Description

      The log-cleaner thread can crash if the number of keys in a topic grows to be too large to fit into the dedupe buffer.

      The result of this is a log line:

      broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner [kafka-log-cleaner-thread-0], Error due to java.lang.IllegalArgumentException: requirement failed: 9750860 messages in segment MY_FAVORITE_TOPIC-2/00000000000047580165.log but offset map can fit only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease log.cleaner.threads

      As a result, the broker is left in a potentially dangerous situation where cleaning of compacted topics is not running.

      It is unclear if the broader strategy for the LogCleaner is the reason for this upper bound, or if this is a value which must be tuned for each specific use-case.

      Of more immediate concern is the fact that the thread crash is not visible via JMX or exposed as some form of service degradation.

      Some short-term remediations we have made are:

      • increasing the size of the dedupe buffer
      • monitoring the log-cleaner threads inside the JVM

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tcrayford-heroku Tom Crayford
                Reporter:
                halorgium Tim Carey-Smith
              • Votes:
                2 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: