[HDDS-11531] Collect iowait and system on the node.

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Ozone Datanode
• Assignee: Shilun Fan
• Reporter: Shilun Fan
• Created: 05/Oct/24 07:31
• Updated: 05/Oct/24 07:32

Description

This feature will periodically retrieve data from /proc/stat, parse the iowait and system information, and write it into metrics. In the future, we will report this metric to SCM so that it can skip high-load DNs when returning the pipeline DN list, decommison data transfers, creating pipelines, and balancing, thereby enhancing the system's access performance.

Parsing `/proc/stat` is a lightweight action that won't impose additional load on the system. This metric has good sensitivity to reflect the current IO performance of the DN. High IOWait typically indicates conditions during high-frequency read/write operations or when using `du` for HDD/SSD access. An increase in System time usually occurs during high-frequency read/write operations with NVMe. When these situations arise, the DN is no longer in optimal condition for data access or writing (as responses will be very slow), so we should find ways to skip such DNs during data access.