[HADOOP-16570] S3A committers leak threads/raises OOM on job/task commit at scale

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.1.2
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 13/Sep/19 11:53
• Updated: 17/Apr/20 15:34
• Resolved: 04/Oct/19 17:57

Description

The fixed size ThreadPool created in AbstractS3ACommitter doesn't get cleaned up at EOL; as a result you leak the no. of threads set in "fs.s3a.committer.threads"

Not visible in MR/distcp jobs, but ultimately causes OOM on Spark

Issue Links

• causes: HADOOP-16650 ITestS3AClosedFS failing -junit test thread
• is related to: HADOOP-16798 job commit failure in S3A MR magic committer test
• links to: GitHub Pull Request #1442