Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4924

Factor out code to launch Spark applications into a separate library

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.4.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      One of the questions we run into rather commonly is "how to start a Spark application from my Java/Scala program?". There currently isn't a good answer to that:

      • Instantiating SparkContext has limitations (e.g., you can only have one active context at the moment, plus you lose the ability to submit apps in cluster mode)
      • Calling SparkSubmit directly is doable but you lose a lot of the logic handled by the shell scripts
      • Calling the shell script directly is doable, but sort of ugly from an API point of view.

      I think it would be nice to have a small library that handles that for users. On top of that, this library could be used by Spark itself to replace a lot of the code in the current shell scripts, which have a lot of duplication.

        Attachments

        1. spark-launcher.txt
          6 kB
          Marcelo Masiero Vanzin

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                vanzin Marcelo Masiero Vanzin
              • Votes:
                3 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: