Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23287

Spark scheduler does not remove initial executor if not one job submitted

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 2.2.1
    • Fix Version/s: None
    • Component/s: Mesos, Scheduler, Spark Core
    • Labels:
    • Environment:

      Description

      When spark application submitted it deploy initial number of executors. If none of job has been submitted to application spark doesn't remove initial executor.

       
      Cluster manager - Mesos 1.4.1

      Spark 2.2.1

      spark app configuration:

      spark.dynamicAllocation.minExecutors=0

      spark.dynamicAllocation.executorIdleTimeout=25s

      spark.dynamicAllocation.initialExecutors=1

      spark.dynamicAllocation.schedulerBacklogTimeout=4s

      spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=5s

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              humanmashine Pavel Plotnikov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: