Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3174

Provide elastic scaling within a Spark application

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.2
    • Fix Version/s: 1.2.0
    • Component/s: Spark Core, YARN
    • Labels:
      None
    • Target Version/s:

      Description

      A common complaint with Spark in a multi-tenant environment is that applications have a fixed allocation that doesn't grow and shrink with their resource needs. We're blocked on YARN-1197 for dynamically changing the resources within executors, but we can still allocate and discard whole executors.

      It would be useful to have some heuristics that

      • Request more executors when many pending tasks are building up
      • Discard executors when they are idle

      See the latest design doc for more information.

        Attachments

        1. dynamic-scaling-executors-10-6-14.pdf
          315 kB
          Andrew Or
        2. SPARK-3174design.pdf
          90 kB
          Sandy Ryza
        3. SparkElasticScalingDesignB.pdf
          196 kB
          Praveen Seluka

          Issue Links

            Activity

              People

              • Assignee:
                andrewor14 Andrew Or
                Reporter:
                sandyr Sandy Ryza
              • Votes:
                0 Vote for this issue
                Watchers:
                45 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: