Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4214

With dynamic allocation, avoid outstanding requests for more executors than pending tasks need

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: Spark Core, YARN
    • Labels:
      None
    • Target Version/s:

      Description

      Dynamic allocation tries to allocate more executors while we have pending tasks remaining. Our current policy can end up with more outstanding executor requests than needed to fulfill all the pending tasks. Capping the executor requests to the number of cores needed to fulfill all pending tasks would make dynamic allocation behavior less sensitive to settings for maxExecutors.

        Attachments

          Activity

            People

            • Assignee:
              sandyr Sandy Ryza
              Reporter:
              sandyr Sandy Ryza
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: