Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7660

Stream Metrics - Memory Analysis

    XMLWordPrintableJSON

Details

    Description

      During the analysis of JVM memory two possible issues were shown which I would like to bring to your attention:
      1) Duplicate strings
      Top findings: 
      string_content="stream-processor-node-metrics" count="534,277"
      string_content="processor-node-id" count="148,437"
      string_content="stream-rocksdb-state-metrics" count="41,832"
      string_content="punctuate-latency-avg" count="29,681" 
       
      "stream-processor-node-metrics"  seems to be used in Sensors.java as a literal and not interned.
       
      2) The HashMap parentSensors from org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl was reported multiple times as suspicious for potentially keeping alive a lot of objects. In our case the reported size was 40-50MB each.
      I haven't looked too deep in the code but noticed that the class Sensor.java which is used as a key in the HashMap does not implement equals or hashCode method. Not sure this is a problem though.
       
      The analysis was done with Dynatrace 7.0
      We are running Confluent 5.0/Kafka2.0-cp1 (Brokers as well as Clients)
       
      Screenshots are attached.

      Attachments

        1. Mem_Collections.jpeg
          222 kB
          Patrik Kleindl
        2. Mem_Hotspots.jpeg
          254 kB
          Patrik Kleindl
        3. Mem_DuplicateStrings.jpeg
          324 kB
          Patrik Kleindl
        4. Mem_KeepAliveSet.jpeg
          491 kB
          Patrik Kleindl
        5. Mem_DuplicateStrings2.jpeg
          897 kB
          Patrik Kleindl
        6. Mem_References.jpeg
          1.21 MB
          Patrik Kleindl
        7. heapdump-1543441898901.hprof
          15.02 MB
          Patrik Kleindl

        Activity

          People

            vvcephei John Roesler
            pkleindl Patrik Kleindl
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: