[HADOOP-18442] Remove the hadoop-openstack module

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: build, fs, fs/swift
• Labels: pull-request-available
• Target Version/s: 3.3.5
• Hadoop Flags: Incompatible change

Release Note

The swift:// connector for openstack support has been removed. It had fundamental problems (swift's handling of files > 4GB). A subset of the S3 protocol is now exported by almost all object store services -please use that through the s3a connector instead. The hadoop-openstack jar remains, only now it is empty of code. This is to ensure that projects which declare the JAR a dependency will still have successful builds.

• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 05/Sep/22 13:41
• Updated: 11/May/23 09:00
• Resolved: 07/Oct/22 11:32

Description

the openstack module doesn't get tested or maintained; it's just something else to keep up to date security wise. As nobody ever files bugs on it it is clearly not being used either.

On-prem object stores support the S3 APIs and/or provide their own hadoop connectors (ozone, IBM).

Let's just cut it completely. As someone who co-authored a lot of it I am happy to do the duty. I will do a quick review of all test to see if there are any left which we could pull into hadoop common...the FS contract tests were initially derived from the ones I did here.

Issue Links

• is related to: SPARK-43448 Remove dummy hadoop-openstack
• links to: GitHub Pull Request #4855
• links to: GitHub Pull Request #4975