Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-16226

Java client: Performance regression in Trogdor benchmark with high partition counts

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.7.0, 3.6.1
    • 3.6.2, 3.8.0, 3.7.1
    • clients

    Description

      Background

      https://issues.apache.org/jira/browse/KAFKA-15415 implemented optimisation in java-client to skip backoff period if client knows of a newer leader, for produce-batch being retried.

      What changed

      The implementation introduced a regression noticed on a trogdor-benchmark running with high partition counts(36000!).
      With regression, following metrics changed on the produce side.

      1. record-queue-time-avg: increased from 20ms to 30ms.
      2. request-latency-avg: increased from 50ms to 100ms.

      Why it happened

      As can be seen from the original PR RecordAccmulator.partitionReady() & drainBatchesForOneNode() started using synchronised method Metadata.currentLeader(). This has led to increased synchronization between KafkaProducer's application-thread that call send(), and background-thread that actively send producer-batches to leaders.

      Lock profiles clearly show increased synchronisation in KAFKA-15415 PR(highlighted in Red) Vs baseline ( see below ). Note the synchronisation is much worse for paritionReady() in this benchmark as its called for each partition, and it has 36k partitions!

      Lock Profile: Kafka-15415

      Lock Profile: Baseline

      Fix

      Synchronization has to be reduced between 2 threads in order to address this. https://github.com/apache/kafka/pull/15323 is a fix for it, as it avoids using Metadata.currentLeader() instead rely on Cluster.leaderFor().

      With the fix, lock-profile & metrics are similar to baseline.

       

      Attachments

        1. baseline_lock_profile.png
          334 kB
          Mayank Shekhar Narula
        2. kafka_15415_lock_profile.png
          343 kB
          Mayank Shekhar Narula

        Issue Links

          Activity

            People

              mayanksnarula Mayank Shekhar Narula
              mayanksnarula Mayank Shekhar Narula
              Walker Carlson Walker Carlson
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: