Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-545

Add a Performance Suite for the Log subsystem

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.8.0
    • 0.8.0
    • None

    Description

      We have had several performance concerns or potential improvements for the logging subsystem. To conduct these in a data-driven way, it would be good to have a single-machine performance test that isolated the performance of the log.

      The performance optimizations we would like to evaluate include

      • Special casing appends in a follower which already have the correct offset to avoid decompression and recompression
      • Memory mapping either all or some of the segment files to improve the performance of small appends and lookups
      • Supporting multiple data directories and avoiding RAID

      Having a standalone tool is nice to isolate the component and makes profiling more intelligible.

      This test would drive load against Log/LogManager controlled by a set of command line options. These command line program could then be scripted up into a suite of tests that covered variations in message size, message set size, compression, number of partitions, etc.

      Here is a proposed usage for the tool:

      ./bin/kafka-log-perf-test.sh
      Option Description
      ------ -----------
      --partitions The number of partitions to write to
      --dir The directory in which to write the log
      --message-size The size of the messages
      --set-size The number of messages per write
      --compression Compression alg
      --messages The number of messages to write
      --readers The number of reader threads reading the data

      The tool would capture latency and throughput for the append() and read() operations.

      Attachments

        1. KAFKA-545.patch
          37 kB
          Jay Kreps
        2. KAFKA-545-draft.patch
          35 kB
          Jay Kreps
        3. KAFKA-545-v2.patch
          39 kB
          Jay Kreps
        4. KAFKA-545-v3.patch
          39 kB
          Jay Kreps

        Activity

          People

            jkreps Jay Kreps
            jkreps Jay Kreps
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: