[SPARK-26941] incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core, YARN
• Assignee: liupengcheng
• Reporter: liupengcheng
• Created: 20/Feb/19 11:26
• Updated: 17/Mar/19 00:47
• Resolved: 17/Mar/19 00:45

Description

Currently, when enabled streaming dynamic allocation for streaming applications, the maxNumExecutorFailures in ApplicationMaster is still computed with `spark.dynamicAllocation.maxExecutors`. 

Actually, we should consider `spark.streaming.dynamicAllocation.maxExecutors` instead.

Related codes:

private val maxNumExecutorFailures = {
  val effectiveNumExecutors =
    if (Utils.isStreamingDynamicAllocationEnabled(sparkConf)) {
      sparkConf.get(STREAMING_DYN_ALLOCATION_MAX_EXECUTORS)
    } else if (Utils.isDynamicAllocationEnabled(sparkConf)) {
      sparkConf.get(DYN_ALLOCATION_MAX_EXECUTORS)
    } else {
      sparkConf.get(EXECUTOR_INSTANCES).getOrElse(0)
    }
  // By default, effectiveNumExecutors is Int.MaxValue if dynamic allocation is enabled. We need
  // avoid the integer overflow here.
  val defaultMaxNumExecutorFailures = math.max(3,
    if (effectiveNumExecutors > Int.MaxValue / 2) Int.MaxValue else (2 * effectiveNumExecutors))

  sparkConf.get(MAX_EXECUTOR_FAILURES).getOrElse(defaultMaxNumExecutorFailures)

Issue Links

• links to: GitHub Pull Request #23845