[HADOOP-14890] Move up to AWS SDK 1.11.199

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0-beta1
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: build, fs/s3
• Target Version/s: 2.9.0, 3.0.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 21/Sep/17 14:00
• Updated: 23/Mar/18 11:45
• Resolved: 26/Sep/17 19:11

Description

the AWS SDK in Hadoop 3.0.-beta-1 prints a warning whenever you call abort() on a stream, which is what we need to do whenever doing long-distance seeks in a large file opened with fadvise=normal

2017-09-20 17:51:50,459 [ScalaTest-main-running-S3ASeekReadSuite] INFO  s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - 
2017-09-20 17:51:50,460 [ScalaTest-main-running-S3ASeekReadSuite] INFO  s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Starting read() [pos = 45603305]
2017-09-20 17:51:50,461 [ScalaTest-main-running-S3ASeekReadSuite] WARN  internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
2017-09-20 17:51:51,263 [ScalaTest-main-running-S3ASeekReadSuite] INFO  s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Duration of read() [pos = 45603305] = 803,650,637 nS

This goes away if we upgrade to the latest SDK, at least for the non-localdynamo bits

Issue Links

• is depended upon by: HADOOP-9991 Fix up Hadoop POMs, roll up JARs to latest versions
• is related to: HADOOP-15040 Upgrade AWS SDK to 1.11.271: NPE in 1.11.199 bug spams logs w/ Yarn Log Aggregation
• relates to: HADOOP-14596 AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error
• relates to: HADOOP-14040 Use shaded aws-sdk uber-JAR 1.11.86