Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23637

Yarn might allocate more resource if a same executor is killed multiple times.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.1, 2.4.0
    • Component/s: YARN
    • Labels:
      None

      Description

      YarnAllocator uses numExecutorsRunning to track the number of running executor. numExecutorsRunning is used to check if there're executors missing and need to allocate more.

       In current code, numExecutorsRunning can be negative when driver asks to kill a same idle executor multiple times.

        Attachments

          Activity

            People

            • Assignee:
              jinxing6042@126.com jin xing
              Reporter:
              jinxing6042@126.com jin xing
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: