[HADOOP-17271] S3A statistics to support IOStatistics

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.3.1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 18/Sep/20 12:30
• Updated: 05/Oct/22 10:31
• Resolved: 31/Dec/20 21:56

Description

S3A to rework statistics with

• API + Implementation split of the interfaces used by subcomponents when reporting stats
• S3A Instrumentation to implement all the interfaces
• streams, etc to all implement IOStatisticsSources and serve to callers
• Add some tracking of durations of remote requests
• RemoteIterator iterators returned from list* calls return list stats for callers to collect; toString will print them too

Issue Links

• breaks: HADOOP-17451 Intermittent failure of S3A tests which make assertions on statistics/IOStatistics
• breaks: HADOOP-17456 S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure
• causes: HADOOP-17774 bytesRead FS statistic showing twice the correct value in S3A
• contains: HADOOP-13845 s3a to instrument duration of HTTP calls
• contains: HADOOP-15348 S3A Input Stream bytes read counter isn't getting through to StorageStatistics/instrumentation properly
• contains: HADOOP-16133 S3A statistic collection underrecords bytes written in helper threads
• fixes: HADOOP-16666 typo in TestNeworkBinding
• links to: GitHub Pull Request #2324
• links to: GitHub Pull Request #2553
• links to: GitHub Pull Request #2580