[KAFKA-15601] Client metrics and observability

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: admin, clients, consumer, core, producer , streams
• Labels: kip
• Assignee: Apoorv Mittal
• Reporter: Apoorv Mittal
• Created: 12/Oct/23 22:27
• Updated: 22/Nov/23 11:32

Description

This Jira tracks the development of KIP-714: https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability

 

Issue Links

• mentioned in: Page Loading...

Sub-Tasks

• Add Telemetry RPCs Definitions: Resolved
• Add Client API in interfaces and client configurations.: Resolved
• Define interfaces/classes for capturing telemetry metrics: Resolved
• Define client telemetry states and their transitions: Resolved
• Implement MetricsCollector which collects metrics in OTLP format: Resolved
• Implement support for clientInstanceIds in Kafka Stream: Resolved
• Implement ClientTelemetryReporter which manages telemetry lifecyclye: Resolved
• Add Opentelemetry Proto library with shadowed classes: Resolved
• Implement telemetry naming strategy: Resolved
• Define new client_metrics resource type configuration to store metric config.: Resolved
• Add support for re2/j regex parsing for patterns: Open
• Add support of client-metrics in kafka-configs.sh: Resolved
• Delete subscription from metadata when all configs are deleted: Resolved
• Add support to describe all subscriptions through utility: Resolved
• Implement ClientMetricsManager to process request: Resolved
• Define broker configurations and exceptions: Resolved
• Investigate new test case failure - testReplicateSourceDefault - MirrorConnectorsIntegrationBaseTest: Resolved
• Implement request and response handling for telemetry APIs: Resolved
• Add support for compression/decompression of metrics: Resolved
• implement capturing client port information from Socket Server: Resolved
• Add support to send only sparse metrics, only non-zero values: Open
• Improve implementation of client instance cache: Resolved
• Add request/response handling in KafkaApis and update metrics plugin: Resolved
• Improve KafkaMetrics APIs to expose Value Provider information: Resolved
• Handle push telemetry throttling with quota manager: Reopened
• Implement kafka-client-metrics.sh tool: Resolved
• Support change of temporality in Java client: Resolved
• Implement client changes to support telemetry APIs: Resolved
• Create public doc for telemetry state transition: Resolved
• Add tests to validate telemetry requests with different version: Resolved
• Add support to exclude labels in client telemetry: Resolved
• Implement broker metrics for client telemetry usage: Resolved
• Fix broker metrics to follow kebab/hyphen case: Resolved
• Integrate metric measurability changes in metrics collector: Resolved
• May be record metrics on UnknownSubscriptionIdException: Open