Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6569

Reflection in OffsetIndex and TimeIndex construction

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: core
    • Labels:
      None

      Description

      kafka.log.AbstractIndex uses the Logging mixin to lazily initialize loggers for any concrete type that inherits from it. This works great, except that the LazyLogging trait uses reflection to compute the logger name.

      When you have hundreds of thousands of log segments to load on startup the extra cost adds up.

      I've attached flame graphs from broker startup on a broker that has about 12TB of log segments to load, and a second flame graph after changing AbstractIndex to statically initialize a logger.

        Attachments

        1. before.png
          532 kB
          Kyle Ambroff-Kao
        2. after.png
          420 kB
          Kyle Ambroff-Kao

          Issue Links

            Activity

              People

              • Assignee:
                ambroff Kyle Ambroff-Kao
                Reporter:
                ambroff Kyle Ambroff-Kao
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: