[HADOOP-18920] RPC Metrics : Optimize logic for log slow RPCs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: metrics
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Haiyang Hu
• Reporter: Haiyang Hu
• Created: 04/Oct/23 07:53
• Updated: 27/Jan/24 06:56
• Resolved: 25/Oct/23 05:59

Description

HADOOP-12325 implement a capability where "slow" RPCs are logged in NN log.
Current processing logic is the "slow" RPCs are to be those whose processing time is outside 3 standard deviation.
However, in practice it is found that many logs of slow rpc are currently output, and sometimes RPCs with a processing time of 1ms are also declared as slow, this is not in line with actual expectations.

Therefore, consider optimize the logic conditions of slow RPC and add a `logSlowRPCThresholdMs` variable to judge whether the current RPCas slow so that the expected slow RPC log can be logger.
for `logSlowRPCThresholdMs`, we can support dynamic refresh to facilitate adjustments based on the actual operating conditions of the hdfs cluster.

Issue Links

• links to: GitHub Pull Request #6146