[SPARK-42425] spark-hadoop-cloud is not provided in the default Spark distribution

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.3.1
• Component/s: Input/Output
• Assignee: Unassigned
• Reporter: Arseniy Tashoyan
• Created: 13/Feb/23 21:46
• Updated: 11/Mar/23 18:03
• Resolved: 10/Mar/23 17:15

Description

The library spark-hadoop-cloud is absent in the default Spark distribution (as well as its dependencies like hadoop-aws). Therefore the dependency management section described in Integration with Cloud Infrastructures is invalid. Actually the libraries for cloud integration are not provided.

A naive workaround would be to add the spark-hadoop-cloud library as a compile-scope dependency. However, this does not work due to Spark classpath hierarchy. Spark system classloader does not see classes loaded by the application classloader.

Therefore a proper fix would be to enable the hadoop-cloud build profile by default: -Phadoop-cloud