[HDFS-16595] Slow peer metrics - add median, mad and upper latency limits

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: metrics
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.5
• Release Note: Namenode metrics that represent Slownode Json now include three important factors (median, median absolute deviation, upper latency limit) that can help user determine how urgently a given slownode requires manual intervention.
• Assignee: Viraj Jasani
• Reporter: Viraj Jasani
• Created: 26/May/22 02:22
• Updated: 27/Jan/24 01:25
• Resolved: 03/Jun/22 22:11

Description

Slow datanode metrics include slow node and it's reporting node details. With HDFS-16582, we added the aggregate latency that is perceived by the reporting nodes.

In order to get more insights into how the outlier slownode's latencies differ from the rest of the nodes, we should also expose median, median absolute deviation and the calculated upper latency limit details.

Issue Links

• is related to: HDFS-16521 DFS API to retrieve slow datanodes
• is related to: HDFS-16582 Expose aggregate latency of slow node as perceived by the reporting node
• links to: GitHub Pull Request #4357
• links to: GitHub Pull Request #4405