[HADOOP-13065] Add a new interface for retrieving FS and FC Statistics

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fs
• Target Version/s: 2.8.0
• Assignee: Mingliang Liu
• Reporter: Ram Venkatesh
• Created: 16/Mar/16 21:05
• Updated: 13/Jul/18 15:20
• Resolved: 11/May/16 21:28

Description

Currently FileSystem.Statistics exposes the following statistics:
BytesRead
BytesWritten
ReadOps
LargeReadOps
WriteOps

These are in-turn exposed as job counters by MapReduce and other frameworks. There is logic within DfsClient to map operations to these counters that can be confusing, for instance, mkdirs counts as a writeOp.

Proposed enhancement:
Add a statistic for each DfsClient operation including create, append, createSymlink, delete, exists, mkdirs, rename and expose them as new properties on the Statistics object. The operation-specific counters can be used for analyzing the load imposed by a particular job on HDFS.
For example, we can use them to identify jobs that end up creating a large number of files.

Once this information is available in the Statistics object, the app frameworks like MapReduce can expose them as additional counters to be aggregated and recorded as part of job summary.

Issue Links

• breaks: HDFS-10418 NPE in TestDistributedFileSystem.testDFSCloseOrdering
• relates to: HADOOP-13028 add low level counter metrics for S3A; use in read performance tests
• relates to: HADOOP-13171 Add StorageStatistics to S3A; instrument some more operations
• relates to: HADOOP-15124 Slow FileSystem.Statistics counters implementation
• relates to: TEZ-3331 Add operation specific HDFS counters for Tez UI

Sub-Tasks

• FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps: Resolved
• Support reset operation for new global storage statistics and per FS storage stats: Resolved
• FileSystem#initialize must not attempt to create StorageStatistics objects with null or empty schemes: Resolved
• FileSystemStorageStatistics must not attempt to read non-existent rack-aware read stats in branch-2.8: Resolved
• Guard null stats key in FileSystemStorageStatistics: Resolved
• Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented: Resolved
• Define common statistics names across schemes: Resolved
• DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation: Resolved
• Use GlobalStorageStatistics.INSTANCE.reset() at FileSystem#clearStatistics(): Resolved