[SPARK-28762] Read JAR main class if JAR is not located in local file system

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Deploy, Spark Core, Spark Submit
• Assignee: Ivan Gozali
• Reporter: Ivan Gozali
• Created: 16/Aug/19 19:04
• Updated: 17/Oct/19 21:38
• Resolved: 17/Oct/19 21:36

Description

Currently, spark-submit doesn't attempt to read the main class from a Spark app JAR file if the scheme of the primary resource URI is not file. In other words, if the JAR is not in the local file system, it will barf.

It would be useful to have this feature if I deploy my Spark app JARs in S3 or HDFS.

If it makes sense to maintainers, I can take a stab at this - I think I know which files to look at.

Issue Links

• links to: GitHub Pull Request #25910