Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7652

Kafka Streams Session store performance degradation from 0.10.2.2 to 0.11.0.0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0.0, 0.11.0.1, 0.11.0.2, 0.11.0.3, 1.1.1, 2.0.0, 2.0.1
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:

      Description

      I'm creating this issue in response to Guozhang Wang's request on the mailing list:

      https://lists.apache.org/thread.html/97d620f4fd76be070ca4e2c70e2fda53cafe051e8fc4505dbcca0321@%3Cusers.kafka.apache.org%3E

      We are attempting to upgrade our Kafka Streams application from 0.10.2.1 but experience a severe performance degradation. The highest amount of CPU time seems spent in retrieving from the local cache. Here's an example thread profile with 0.11.0.0:

      https://i.imgur.com/l5VEsC2.png

      When things are running smoothly we're gated by retrieving from the state store with acceptable performance. Here's an example thread profile with 0.10.2.1:

      https://i.imgur.com/IHxC2cZ.png

      Some investigation reveals that it appears we're performing about 3 orders magnitude more lookups on the NamedCache over a comparable time period. I've attached logs of the NamedCache flush logs for 0.10.2.1 and 0.11.0.3.

      We're using session windows and have the app configured for commit.interval.ms = 30 * 1000 and cache.max.bytes.buffering = 10485760

      I'm happy to share more details if they would be helpful. Also happy to run tests on our data.

      I also found this issue, which seems like it may be related:

      https://issues.apache.org/jira/browse/KAFKA-4904

       

      KIP-420: https://cwiki.apache.org/confluence/display/KAFKA/KIP-420%3A+Add+Single+Value+Fetch+in+Session+Stores 

       

        Attachments

        1. kafka_10_2_1_flushes.txt
          5 kB
          Jonathan Gordon
        2. kafka_11_0_3_flushes.txt
          5 kB
          Jonathan Gordon
        3. 2.2.0-rc0_b-NamedCache.txt
          3 kB
          Jonathan Gordon
        4. 0.10.2.1-NamedCache.txt
          3 kB
          Jonathan Gordon
        5. 2.3.0-7652-NamedCache.txt
          3 kB
          Jonathan Gordon

          Issue Links

            Activity

              People

              • Assignee:
                guozhang Guozhang Wang
                Reporter:
                jonathanpdx Jonathan Gordon
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: