[HBASE-28302] Add tracking of fs read times in ScanMetrics and slow logs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-beta-2

Release Note

Adds a new getFsReadTime() to the slow log records, and fsReadTime counter to ScanMetrics. In both cases, this is the cumulative time spent reading blocks from hdfs for the given request. Additionally, a new fsSlowReadsCount jmx metric is added to the sub=IO bean. This is the count of HDFS reads which took longer than hbase.fs.reader.warn.time.ms.

• Assignee: Bryan Beaudreault
• Reporter: Bryan Beaudreault
• Created: 10/Jan/24 14:43
• Updated: 25/Jan/24 02:01
• Resolved: 23/Jan/24 18:10

Description

We've had this in our production for a while, and it's useful info to have. We already track FS read times in HFileBlock. We can project that into the ScanMetrics instance and slow log pretty easily. It is also helpful to add a slow.fs.read.threshold, over which we log a warn

Issue Links

• links to: GitHub Pull Request #5622