Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35317

Job submission in high performance computing environments

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.1.2
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
      None

      Description

      Spark is often used in high performance computing environments. The default launcher does not directly support schedulers used such as slurm and pbs.  It would be good to support these directly. The repositories https://github.com/ekasitk/spark-on-hpc and https://github.com/rokroskar/sparkhpc contain most of the material necessary for this, but it may be good to incorporate this in Spark directly so that Java and Scala are also well supported.

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment