[HADOOP-13826] S3A Deadlock in multipart copy due to thread pool limits.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.3
• Fix Version/s: 2.8.0, 3.0.0-alpha4
• Component/s: fs/s3
• Target Version/s: 2.8.0
• Assignee: Sean Mackrory
• Reporter: Sean Mackrory
• Created: 21/Nov/16 20:15
• Updated: 21/Apr/17 21:01
• Resolved: 21/Feb/17 18:30

Description

In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The TransferManager javadocs (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html) explain how this is possible:

It is not recommended to use a single threaded executor or a thread pool with a bounded work queue as control tasks may submit subtasks that can't complete until all sub tasks complete. Using an incorrectly configured thread pool may cause a deadlock (I.E. the work queue is filled with control tasks that can't finish until subtasks complete but subtasks can't execute because the queue is filled).

Issue Links

• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance