[SPARK-35317] Job submission in high performance computing environments

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.1.2
• Component/s: Spark Submit
• Assignee: Unassigned
• Reporter: Benson Muite
• Created: 05/May/21 06:56
• Updated: 12/Dec/22 18:10
• Resolved: 12/May/21 07:38

Description

Spark is often used in high performance computing environments. The default launcher does not directly support schedulers used such as slurm and pbs.  It would be good to support these directly. The repositories https://github.com/ekasitk/spark-on-hpc and https://github.com/rokroskar/sparkhpc contain most of the material necessary for this, but it may be good to incorporate this in Spark directly so that Java and Scala are also well supported.