[HADOOP-14132] Filesystem discovery to stop loading implementation classes

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.3
• Component/s: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 28/Feb/17 11:17
• Updated: 04/Jan/24 12:03

Description

Integration testing of Hadoop with the HADOOP-14040 has shown up that the move to a shaded AWS JAR is slowing all hadoop client code down.

I believe this is due to how we use service discovery to identify FS implementations: the implementation classes themselves are instantiated.
This has known problems today with classloading, but clearly impacts performance too, especially with complex transitive dependencies unique to the loaded class.

Proposed: have lightweight service declaration classes which implement an interface declaring

1. schema
2. classname of FileSystem impl
3. classname of AbstractFS impl
4. homepage (for third party code, support, etc)

These are what we register and scan in the FS to look for services.

This will leave the question about what to do for existing filesystems? I think we'll need to retain the old code for external ones, while moving the hadoop modules to the new ones

Issue Links

• depends upon: HADOOP-14123 Remove misplaced ADL service provider config file for FileSystem
• depends upon: HADOOP-14183 Remove service loader config file for wasb fs
• depends upon: HADOOP-13606 swift FS to add a service load metadata file
• depends upon: HADOOP-14184 Remove service loader config entry for ftp fs
• depends upon: HADOOP-14185 Remove service loader config entry for Har fs
• is related to: HADOOP-16102 FilterFileSystem does not implement getScheme
• is related to: HADOOP-17402 Add GCS FS impl reference to core-default.xml
• relates to: HADOOP-14138 Remove S3A ref from META-INF service discovery, rely on existing core-default entry