Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3821

Develop an automated way of creating Spark images (AMI, Docker, and others)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • Build, EC2
    • None

    Description

      Right now the creation of Spark AMIs or Docker containers is done manually. With tools like Packer, we should be able to automate this work, and do so in such a way that multiple types of machine images can be created from a single template.

      Attachments

        1. packer-proposal.html
          21 kB
          Nicholas Chammas

        Issue Links

          Is contained by

          Umbrella - An overarching type made of sub-tasks SPARK-4325 Improve spark-ec2 cluster launch times

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue SPARK-4958 Bake common tools like ganglia into Spark AMI

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-4241 spark_ec2.py support China AWS region: cn-north-1

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. SPARK-922 Update Spark AMI to Python 2.7

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-3314 Script creation of AMIs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link PR to add Packer scripts

          Web Link Proposed implementation with Packer (nchammas)

          Web Link Related dev list discussion: "EC2 clusters ready in launch time + 30 seconds"

          Activity

            People

              nchammas Nicholas Chammas
              nchammas Nicholas Chammas
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: