[HADOOP-14283] Upgrade AWS SDK to 1.11.134

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha2
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Component/s: fs/s3
• Hadoop Flags: Reviewed
• Assignee: Aaron Fabbri
• Reporter: Aaron Fabbri
• Created: 05/Apr/17 22:10
• Updated: 19/Jan/19 17:27
• Resolved: 08/Jun/17 22:08

Description

We hit a hang bug when testing S3A with parallel renames.

I narrowed this down to the newer AWS Java SDK. It only happens under load, and appears to be a failure to wake up a waiting thread on timeout/error.

I've created a github issue here:
https://github.com/aws/aws-sdk-java/issues/1102

I can post a Hadoop scale test which reliably reproduces this after some cleanup. I have posted an SDK-only test here which reproduces the issue without Hadoop:

https://github.com/ajfabbri/awstest

I have a support ticket open and am working with Amazon on this bug so I'll take this issue.

Issue Links

• breaks: HADOOP-14589 LocalDynamoDB missing from latest AWS SDK releases
• is depended upon by: HADOOP-9991 Fix up Hadoop POMs, roll up JARs to latest versions
• is depended upon by: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• is duplicated by: HADOOP-14596 AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error
• is related to: HADOOP-16055 Upgrade AWS SDK to 1.11.271 in branch-2