[HADOOP-13551] Collect AwsSdkMetrics in S3A FileSystem IOStatistics

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0-beta1
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.3.1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 26/Aug/16 20:49
• Updated: 29/Nov/22 15:37
• Resolved: 25/Mar/21 14:02

Description

The S3A Connector has the ability to pass statistics collected by the AWS SDK into the IOStatistics store of the (stream, FS)

But

• wiring up doesn't (yet) work
• its best if there was thread-context level collection, though FS-level statistics would be a good start.

Issue Links

• blocks: HADOOP-17469 IOStatistics Phase II
• causes: HADOOP-17771 S3AFS creation fails "Unable to find a region via the region provider chain."
• contains: HADOOP-17372 S3A AWS Credential provider loading gets confused with isolated classloaders
• fixes: HADOOP-17673 IOStatistics API in branch-3.3 break compatibility
• is blocked by: HADOOP-16830 Add Public IOStatistics API
• is duplicated by: HADOOP-13553 hook up AwsSdkMetrics to hadoop metrics
• is duplicated by: HADOOP-15055 Add s3 metrics from AWS SDK to s3a metrics tracking
• supercedes: HADOOP-17497 refactoring the signature of S3ClientFactory breaks hboss compile
• links to: GitHub Pull Request #2687
• links to: GitHub Pull Request #2778
• links to: GitHub Pull Request #2814
• links to: GitHub Pull Request #2815