[HADOOP-18993] S3A: Add option fs.s3a.classloader.isolation (#6301)

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.6
• Fix Version/s: 3.4.0, 3.5.0, 3.4.1
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.5.0
• Release Note: If the user wants to load custom implementations of AWS Credential Providers through user provided jars can set {{fs.s3a.extensions.isolated.classloader}} to {{false}}.

Language

java

• Assignee: Antonio Murgia
• Reporter: Antonio Murgia
• Created: 27/Nov/23 21:56
• Updated: 11/Feb/24 01:08
• Resolved: 05/Feb/24 18:03

Description

In HADOOP-17372 the S3AFileSystem forces the configuration classloader to be the same as the one that loaded S3AFileSystem. This leads to the impossibility in Spark applications to load third party credentials providers as user jars.

The option fs.s3a.classloader.isolation (default: true) can be set to false to disable s3a classloader isolation;

This can assist in using custom credential providers and other extension points.

Issue Links

• links to: GitHub Pull Request #6301