[HIVE-18861] druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Druid integration
• Assignee: Steve Loughran Assign to me
• Reporter: Steve Loughran
• Created: 05/Mar/18 15:21
• Updated: 22/May/18 23:13
• Resolved: 08/Mar/18 17:28

Description

druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP, even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.

Two options

1. exclude the dependency
2. force it up to whatever ${hadoop.version} is, so make it consistent

Issue Links

• relates to: HIVE-16276 Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V