[HADOOP-16133] S3A statistic collection underrecords bytes written in helper threads

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Done
• Affects Version/s: 3.1.2
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 21/Feb/19 19:51
• Updated: 06/Jan/21 16:06
• Resolved: 06/Jan/21 16:06

Description

Applications collecting per-thread statistics from S3A get underreporting of bytes written, as all byte written in the worker call update those in a different thread.

Proposed:

• the bytes upload statistics are uploaded in the primary thread as a block is queued for write, not after in the completion phase in the other thread
• final WriteOperationsHelper.writeSuccessful() takes the final statistics for its own entertainment

Really I want context-specific storage statistics.

Issue Links

• Is contained by: HADOOP-17271 S3A statistics to support IOStatistics