[HADOOP-18650] improve s3a committer stats collected

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.5
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 03/Mar/23 18:22
• Updated: 03/Apr/23 13:01

Description

we can improve stats collected in the s3a committer and saved to the JSON.

key ones

1. of task manifests read; duration of loads
2. size of each manifest

I think we would also benefit if we could set the commit thread pools to be big -but then shared across all jobs (i.e. demand-created thread pool in s3a fs). that would allow for a pool size of say, 500, but still support many jobs actively committing at same time (busy spark driver)
finally: should file commit pool size be > size of pool of manifest readers. I think it could be, but the ratio should be fairly low.

Issue Links

• is related to: MAPREDUCE-7435 ManifestCommitter OOM on azure job