[KAFKA-4179] Replication Throttling: Add Usability Metrics PartitionBytesInRate & SumReplicaLag

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.1.0
• Component/s: replication
• Assignee: Unassigned
• Reporter: Ben Stopford
• Created: 16/Sep/16 08:07
• Updated: 16/Sep/16 08:09

Description

Add two new metrics to Kafka

PartitionBytesInRate: Equivalent to BytesInPerSec, but at a partition level (i.e. total traffic - throttled and not throttled). This is required for estimating how long a rebalance will take to complete. B/s. See usability section below.

SumReplicaLag: This is the sum of all replica lag values on the broker. This metric is used to monitor progress of a rebalance and is particularly useful for determining if the rebalance has become stuck due to an overly harsh throttle value (as the metric will stop decreasing).

As covered in KIP-73 https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas

These make it possible for an administrator to calculate how long a rebalance will take.