[FLINK-13044] Shading of AWS SDK in flink-s3-fs-hadoop results in ClassNotFoundExceptions

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.9.0
• Component/s: FileSystems
• Labels: pull-request-available
• External issue URL: https://issues.apache.org/jira/browse/MSHADE-156
• External issue ID: MSHADE-156
• Assignee: Chesnay Schepler
• Reporter: Sebastian J.
• Created: 01/Jul/19 12:40
• Updated: 02/Oct/19 17:44
• Resolved: 06/Aug/19 19:44

Description

 The current build configuration shades the AWS SDK into flink-s3-fs-hadoop.jar

Due to MSHAD-156 the shading itself does not only add the classes to the JAR but also alters the string literal PACKAGE_PREFIXES_TO_SHADE in org.apache.flink.fs.s3hadoop.S3FileSystemFactory from com.amazonaws. to org.apache.flink.fs.s3base.shaded.com.amazonaws - which causes settings like

fs.s3a.aws.credentials.provider: com.amazonaws.auth.DefaultAWSCredentialsProviderChain

to not be remapped properly as the startsWith in org.apache.flink.fs.s3.common.HadoopConfigLoader.shadeClassConfig() doesn't match the beginning of the FQCN.

This is a known issue in the Maven shade plugin (see MSHAD-156) - a possible workaround is named in the ticket, too.

This issue is only visible looking into the compiled class file after the shading happened!

Fixing the PACKAGE_PREFIXES_TO_SHADE value alone doesn't help as based on FLINK_SHADING_PREFIX the FQCN is assembled as

org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain

which can't be found either;
the class is located at

org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.DefaultAWSCredentialsProviderChain

Issue Links

• is broken by: MSHADE-156 shade plugin is transforming also strings that are not supposed to be transformed
• relates to: FLINK-13602 S3 Presto filesystem effectively does not support credential providers
• relates to: FLINK-8439 Add Flink shading to AWS credential provider s3 hadoop config
• links to: GitHub Pull Request #9071
• links to: GitHub Pull Request #9318