[KAFKA-16129] Add integration test for KIP-977

• Type: Task
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Qichao Chu
• Created: 14/Jan/24 22:00
• Updated: 14/Jan/24 22:00

Description

metrics.verbosity will be a new dynamic config introduced to control the verbosity(fan-out rate) of the metrics. It's a config with JSON format specifying the condition controlling fan-out of the metrics. If the value high is set for the level key of the configured JSON(see below for example values), high fan-out tags(e.g. partition)will be added to metrics specified by the name filter and will apply to all the topics that meet the conditions in the filters section. In the low settings, these tags will be assigned with an empty value. We elected to make it central so that this implementation can be generalized in the future either into a library, or allow other means for centralized control.

More details: https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics

 

The following 3 tests will be done for common metrics collectors: JMX, Prometheus, and OpenTelemetry.

1. The partition tag can be observed from metrics if high verbosity is used
2. The partition tag should result in an empty string or be filtered out by the metrics collector if default verbosity is used
3. Dynamically setting the verbosity can result in the behavior defined in the above tests

Issue Links

• is a child of: KAFKA-15447 Partition Level Throughput Metrics