[KAFKA-3811] Introduce Kafka Streams metrics recording levels

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: streams
• Assignee: Eno Thereska
• Reporter: Greg Fodor
• Created: 09/Jun/16 17:59
• Updated: 05/Dec/16 17:10
• Resolved: 05/Dec/16 17:10

Description

Follow-up from the discussions here:

https://github.com/apache/kafka/pull/1447
https://issues.apache.org/jira/browse/KAFKA-3769

The proposal is to introduce configuration to control the granularity/volumes of metrics emitted by Kafka Streams jobs, since the per-record level metrics introduce non-trivial overhead and are possibly less useful once a job has been optimized.

Proposal from guozhangwang:

level0 (stream thread global): per-record process / punctuate latency, commit latency, poll latency, etc

level1 (per processor node, and per state store): IO latency, per-record .. latency, forward throughput, etc.

And by default we only turn on level0.

Issue Links

• relates to: KAFKA-3826 Sampling on throughput / latency metrics recording in Streams