Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6175

AbstractIndex should cache index file to avoid unnecessary disk access during resize()

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.1
    • Component/s: None
    • Labels:
      None

      Description

      Currently when we shutdown a broker, we will call AbstractIndex.resize() for all segments on the broker, regardless of whether the log segment is active or not. AbstractIndex.resize() incurs raf.setLength(), which is expensive because it accesses disks. If we do a threaddump during either LogManger.shutdown() or LogManager.loadLogs(), most threads are in RUNNABLE state at java.io.RandomAccessFile.setLength().

      This patch intends to speed up broker startup and shutdown time by skipping AbstractIndex.resize() for inactive log segments.

      Here is the time of LogManager.shutdown() in various settings. In all these tests, broker has roughly 6k partitions and 19k segments.

      • If broker does not have this patch and KAFKA-6172, LogManager.shutdown() takes 69 seconds
      • If broker has KAFKA-6172 but not this patch, LogManager.shutdown() takes 21 seconds.
      • If broker has KAFKA-6172 and this patch, LogManager.shutdown() takes 1.6 seconds.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lindong Dong Lin
                Reporter:
                lindong Dong Lin
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: