[HADOOP-15924] Hadoop aws does not use shaded jars

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.2.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Bharat Viswanadham
• Created: 13/Nov/18 00:18
• Updated: 29/Jan/19 12:21
• Resolved: 13/Nov/18 10:07

Description

Issue is hadoop-aws cannot be used with shaded jars.

The recommended client side jars for hadoop 3 are client-api/runtime shaded jars.
They shade guava etc. So something like SemaphoredDelegatingExecutor refers to shaded guava classes.

hadoop-aws has S3AFileSystem implementation which refers to SemaphoredDelegatingExecutor with unshaded guava ListeningService in the constructor. When S3AFileSystem is created then it uses the hadoop-api jar and finds SemaphoredDelegatingExecutor but not the right constructor because in client-api jar SemaphoredDelegatingExecutor constructor has the shaded guava ListenerService.

So essentially none of the aws/azure/adl hadoop FS implementations will work with the shaded Hadoop client runtime jars.

 

This Jira is created to track the work required to make hadoop-aws work with hadoop shaded client jars.

 

The solution for this can be, hadoop-aws depends on hadoop shaded jars. In this way, we shall not see the issue. Currently, hadoop-aws depends on aws-sdk-bundle and all other remaining jars are provided dependencies.

 

cc steve_l

 

 

Issue Links

• is related to: HADOOP-16080 hadoop-aws does not work with hadoop-client-api
• relates to: HADOOP-15387 Produce a shaded hadoop-cloud-storage JAR for applications to use