Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35317

Job submission in high performance computing environments

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 3.1.2
    • None
    • Spark Submit
    • None

    Description

      Spark is often used in high performance computing environments. The default launcher does not directly support schedulers used such as slurm and pbs.  It would be good to support these directly. The repositories https://github.com/ekasitk/spark-on-hpc and https://github.com/rokroskar/sparkhpc contain most of the material necessary for this, but it may be good to incorporate this in Spark directly so that Java and Scala are also well supported.

      Attachments

        Activity

          People

            Unassigned Unassigned
            baksmj Benson Muite
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: