Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3811

Introduce Kafka Streams metrics recording levels

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams
    • Labels:
      None

      Description

      Follow-up from the discussions here:

      https://github.com/apache/kafka/pull/1447
      https://issues.apache.org/jira/browse/KAFKA-3769

      The proposal is to introduce configuration to control the granularity/volumes of metrics emitted by Kafka Streams jobs, since the per-record level metrics introduce non-trivial overhead and are possibly less useful once a job has been optimized.

      Proposal from guozhangwang:

      level0 (stream thread global): per-record process / punctuate latency, commit latency, poll latency, etc

      level1 (per processor node, and per state store): IO latency, per-record .. latency, forward throughput, etc.

      And by default we only turn on level0.

        Attachments

        1. Muon-Snapshot.zip
          3.85 MB
          Greg Fodor
        2. screenshot-1.png
          205 kB
          Greg Fodor
        3. Muon-latency.zip
          4.23 MB
          Greg Fodor
        4. screenshot-latency.png
          207 kB
          Greg Fodor

          Issue Links

            Activity

              People

              • Assignee:
                enothereska Eno Thereska
                Reporter:
                gfodor Greg Fodor
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: