[HADOOP-17470] Collect more S3A IOStatistics

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.2
• Component/s: fs/s3
• Target Version/s: 3.4.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 14/Jan/21 14:23
• Updated: 25/May/21 14:47
• Resolved: 25/May/21 14:47

Description

collect more stats of

• how long FS API calls are taking
• what is going wrong

API

• Duration of rename(), delete() other common API Calls.
• openFile: how long the open operation is taking, split of open and openfile
• time to initialize FS instance (maybe better in FileSystem.cache?)
• create: Time to complete the create checks
• finishedWrite: time to execute finishedWrite operations (which may include bulk deletes)

Failure tracking: what failures are we seeing as counters
Interrupts, connections, auth failures:

This would be done in S3ARetryPolicy with an IOStatisticsStore passed in.

Issue Links

• Is contained by: HADOOP-17511 Add an Audit plugin point for S3A auditing/context