[HADOOP-16798] job commit failure in S3A MR magic committer test

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Jan/20 16:28
• Updated: 27/Jul/20 10:37
• Resolved: 30/Jun/20 09:54

Description

failure in

ITestS3ACommitterMRJob.test_200_execute:304->Assert.fail:88 Job job_1578669113137_0003 failed in state FAILED with cause Job commit failed: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6e894de2 rejected from org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor@225eed53[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

Stack implies thread pool rejected it, but toString says "Terminated". Race condition?

update 2020-04-22: it's caused when a task is aborted in the AM -the threadpool is disposed of, and while that is shutting down in one thread, task commit is initiated using the same thread pool. When the task committer's destroy operation times out, it kills all the active uploads.

Issue Links

• relates to: HADOOP-16570 S3A committers leak threads/raises OOM on job/task commit at scale
• links to: GitHub Pull Request #1963