[HADOOP-13031] Rack-aware read bytes stats should be managed by HFDS specific StorageStatistics

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs
• Assignee: Unassigned
• Reporter: Mingliang Liu
• Created: 16/Apr/16 02:17
• Updated: 18/Apr/20 06:59

Description

HADOOP-13065 added a new interface for retrieving FS and FC Statistics. This jira is to refactor the code that maintains rack-aware read metrics to use the newly added StorageStatistics. Specially,

1. Rack-aware read bytes metrics is mostly specific to HDFS. For example, local file system doesn't need that. We consider to move it from base FileSystemStorageStatistics to a dedicated HDFS specific StorageStatistics sub-class.
2. We would have to develop an optimized thread-local mechanism to do this, to avoid causing a performance regression in HDFS stream performance.

Optionally, it would be better to simply move this to HDFS's existing per-stream ReadStatistics for now. As HDFS-9579 states, ReadStatistics metrics are only accessible via DFSClient or DFSInputStream. Not something that application framework such as MR and Tez can get to.

Issue Links

• is blocked by: HADOOP-13032 Refactor FileSystem$Statistics to use StorageStatistics
• is related to: HDFS-13992 cross-cluster rack awareness for distcp
• is related to: MAPREDUCE-6660 Add MR Counters for bytes-read-by-network-distance FileSystem metrics
• relates to: HDFS-9579 Provide bytes-read-by-network-distance metrics at FileSystem.Statistics level