Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17566 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-17372

S3A AWS Credential provider loading gets confused with isolated classloaders

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Problem: exception in loading S3A credentials for an FS, "Class class com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement AWSCredentialsProvider"

      Location: S3A + Spark dataframes test

      Hypothesised cause:

      Configuration.getClasses() uses the context classloader, and with the spark isolated CL that's different from the one the s3a FS uses, so it can't load AWS credential providers.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: