[HADOOP-14138] Remove S3A ref from META-INF service discovery, rely on existing core-default entry

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.9.0
• Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha4
• Component/s: fs/s3
• Target Version/s: 2.9.0
• Hadoop Flags: Incompatible change

Release Note

The classpath implementing the s3a filesystem is now defined in core-default.xml. Attempting to instantiate an S3A filesystem instance using a Configuration instance which has not included the default resorts will fail. Applications should not be doing this anyway, as it will lose other critical configuration options needed by the filesystem.

• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 02/Mar/17 11:07
• Updated: 07/Aug/17 10:28
• Resolved: 02/Mar/17 21:46

Description

As discussed in HADOOP-14132, the shaded AWS library is killing performance starting all hadoop operations, due to classloading on FS service discovery.

This is despite the fact that there is an entry for fs.s3a.impl in core-default.xml, we don't need service discovery here

Proposed:

1. cut the entry from /hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
2. when HADOOP-14132 is in, move to that, including declaring an XML file exclusively for s3a entries

I want this one in first as its a major performance regression, and one we coula actually backport to 2.7.x, just to improve load time slightly there too

Issue Links

• is broken by: HADOOP-14040 Use shaded aws-sdk uber-JAR 1.11.86
• is related to: HADOOP-14132 Filesystem discovery to stop loading implementation classes
• relates to: AMBARI-21665 shaded AWS SDK in Hadoop JARs slowing ambari view loading