Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35317

Job submission in high performance computing environments

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.1.2
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
      None

      Description

      Spark is often used in high performance computing environments. The default launcher does not directly support schedulers used such as slurm and pbs.  It would be good to support these directly. The repositories https://github.com/ekasitk/spark-on-hpc and https://github.com/rokroskar/sparkhpc contain most of the material necessary for this, but it may be good to incorporate this in Spark directly so that Java and Scala are also well supported.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              baksmj Benson Muite

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment