[HADOOP-14973] [s3a] Log StorageStatistics

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0-beta1, 2.8.1
• Component/s: fs/s3
• Assignee: Sean Mackrory
• Reporter: Sean Mackrory
• Created: 23/Oct/17 16:19
• Updated: 20/Nov/17 21:16
• Resolved: 20/Nov/17 21:16

Description

S3A is currently storing much more detailed metrics via StorageStatistics than are logged in a MapReduce job. Eventually, it would be nice to get Spark, MapReduce and other workloads to retrieve and store these metrics, but it may be some time before they all do that. I'd like to consider having S3A publish the metrics itself in some form. This is tricky, as S3A has no daemon but lives inside various other processes.

Perhaps writing to a log file at some configurable interval and on close() would be the best we could do. Other ideas would be welcome.

Issue Links

• duplicates: HADOOP-14475 Metrics of S3A don't print out when enable it in Hadoop metrics property file
• is related to: HADOOP-14988 WASB: Expose WASB status metrics as counters in Hadoop
• relates to: HADOOP-14475 Metrics of S3A don't print out when enable it in Hadoop metrics property file