Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16435

Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Scheduler, Spark Core
    • Labels:
      None

      Description

      After SPARK-13723, the behavior changed for spark.dynamicAllocation.initialExecutors less then spark.dynamicAllocation.minExecutors situation.

      initialExecutors < minExecutors is an invalid setting,

      Before SPARK-13723

      If initialExecutors < minExecutors, Spark will throw exception with:

      java.lang.IllegalArgumentException: requirement failed: initial executor number xxx must between min executor number xxx and max executor number xxx
      

      This will clearly let user know that current configuration is invalid.

      After SPARK-13723

      Because we also consider spark.executor.instances, so the initial number is the max value between minExecutors, initialExecutors, numExecutors.

      This will silently ignore the situation where initialExecutors < minExecutors.

      So at least we should add some warning logs to let user know this is an invalid configuration.

      What do you think Thomas Graves, Ryan Blue

        Attachments

          Activity

            People

            • Assignee:
              jerryshao Saisai Shao
              Reporter:
              jerryshao Saisai Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: