[HIVE-16276] Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Druid integration
• Assignee: Jesús Camacho Rodríguez
• Reporter: Sahil Takiar
• Created: 22/Mar/17 00:59
• Updated: 27/Feb/24 22:23
• Resolved: 24/Mar/17 08:40

Description

The druid-handler is pulling in some com.amazonaws dependencies that conflict with the version that Hadoop is using. This causes the above exception to be thrown when running Hive against S3. This patch fixes the dependency issue by shading the aws dependencies in the druid artifacts. Unfortunately, I can't find a great way to add a test for this so it doesn't happen in the future. We will need some more robust S3-integration tests for that.

Issue Links

• is related to: HIVE-18861 druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x