[HADOOP-17372] S3A AWS Credential provider loading gets confused with isolated classloaders

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.4.0
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Nov/20 14:54
• Updated: 27/Nov/23 22:24
• Resolved: 25/May/21 14:46

Description

Problem: exception in loading S3A credentials for an FS, "Class class com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement AWSCredentialsProvider"

Location: S3A + Spark dataframes test

Hypothesised cause:

Configuration.getClasses() uses the context classloader, and with the spark isolated CL that's different from the one the s3a FS uses, so it can't load AWS credential providers.

Issue Links

• Is contained by: HADOOP-13551 Collect AwsSdkMetrics in S3A FileSystem IOStatistics
• is related to: SPARK-37771 Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException
• relates to: SPARK-9206 ClassCastException using HiveContext with GoogleHadoopFileSystem as fs.defaultFS