Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-545

Add a Performance Suite for the Log subsystem



    • Type: New Feature
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:


      We have had several performance concerns or potential improvements for the logging subsystem. To conduct these in a data-driven way, it would be good to have a single-machine performance test that isolated the performance of the log.

      The performance optimizations we would like to evaluate include

      • Special casing appends in a follower which already have the correct offset to avoid decompression and recompression
      • Memory mapping either all or some of the segment files to improve the performance of small appends and lookups
      • Supporting multiple data directories and avoiding RAID

      Having a standalone tool is nice to isolate the component and makes profiling more intelligible.

      This test would drive load against Log/LogManager controlled by a set of command line options. These command line program could then be scripted up into a suite of tests that covered variations in message size, message set size, compression, number of partitions, etc.

      Here is a proposed usage for the tool:

      Option Description
      ------ -----------
      --partitions The number of partitions to write to
      --dir The directory in which to write the log
      --message-size The size of the messages
      --set-size The number of messages per write
      --compression Compression alg
      --messages The number of messages to write
      --readers The number of reader threads reading the data

      The tool would capture latency and throughput for the append() and read() operations.


        1. KAFKA-545-v3.patch
          39 kB
          Jay Kreps
        2. KAFKA-545-v2.patch
          39 kB
          Jay Kreps
        3. KAFKA-545.patch
          37 kB
          Jay Kreps
        4. KAFKA-545-draft.patch
          35 kB
          Jay Kreps



            • Assignee:
              jkreps Jay Kreps
              jkreps Jay Kreps
            • Votes:
              0 Vote for this issue
              2 Start watching this issue


              • Created: