[HDFS-17290] HDFS: add client rpc backoff metrics due to disconnection from lowest priority queue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10.0, 3.4.0
• Fix Version/s: 3.4.0
• Component/s: metrics
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Lei Yang
• Reporter: Lei Yang
• Created: 14/Dec/23 23:36
• Updated: 27/Jan/24 01:08
• Resolved: 16/Jan/24 08:08

Description

Clients are backoff when rpcs cannot be enqueued. However there are different scenarios when backoff could happen. Currently there is no way to differenciate whether a backoff happened due to lowest prio+disconnection or queue overflow from higher priority queues when connection between client and namenode remains open. Currently IPC server just emits a single metrics for all the backoffs.

Example:

1. Client are directly enqueued into lowest priority queue and backoff when lowest queue is full. Client are expected to disconnect from namenode.
2. Client are enqueued into non-lowest priority queue and overflowed all the way down to lowest priority queue and back off. In this case, connection between client and namenode remains open.

We would like to add metrics for #1

Issue Links

• links to: GitHub Pull Request #6359