Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2691

Allow Spark on Mesos to be launched with Docker

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.4.0
    • Mesos

    Description

      Currently to launch Spark with Mesos one must upload a tarball and specifiy the executor URI to be passed in that is to be downloaded on each slave or even each execution depending coarse mode or not.

      We want to make Spark able to support launching Executors via a Docker image that utilizes the recent Docker and Mesos integration work.

      With the recent integration Spark can simply specify a Docker image and options that is needed and it should continue to work as-is.

      Attachments

        1. spark-docker.patch
          5 kB
          Chris Heller

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-8734 Expose all Mesos DockerInfo options to Spark

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link GitHub Pull Request for this feature

          Activity

            People

              ChrisHeller Chris Heller
              tnachen Timothy Chen
              Votes:
              9 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: