[KAFKA-3826] Sampling on throughput / latency metrics recording in Streams

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: streams
• Labels: architecture performance
• Assignee: Unassigned
• Reporter: Guozhang Wang
• Created: 12/Jun/16 02:45
• Updated: 01/Mar/19 22:42
• Resolved: 01/Mar/19 22:42

Description

In Kafka Streams we record throughput / latency metrics on EACH processing record, causing a lot of recording overhead. Instead, we should consider statistically sampling messages flowing through to measures latency and throughput.

This is based on our observations from KAFKA-3769 and KAFKA-3811.

Issue Links

• Is contained by: KAFKA-5152 Kafka Streams keeps restoring state after shutdown is initiated during startup
• is related to: KAFKA-3769 KStream job spending 60% of time writing metrics
• is related to: KAFKA-3811 Introduce Kafka Streams metrics recording levels