[SPARK-46681] Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid unnecessary computations when `MAX_EXECUTOR_FAILURES` is configured

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 11/Jan/24 12:15
• Updated: 12/Jan/24 10:56
• Resolved: 12/Jan/24 10:56

Description

def maxNumExecutorFailures(sparkConf: SparkConf): Int = {
  val effectiveNumExecutors =
    if (Utils.isStreamingDynamicAllocationEnabled(sparkConf)) {
      sparkConf.get(STREAMING_DYN_ALLOCATION_MAX_EXECUTORS)
    } else if (Utils.isDynamicAllocationEnabled(sparkConf)) {
      sparkConf.get(DYN_ALLOCATION_MAX_EXECUTORS)
    } else {
      sparkConf.get(EXECUTOR_INSTANCES).getOrElse(0)
    }
  // By default, effectiveNumExecutors is Int.MaxValue if dynamic allocation is enabled. We need
  // avoid the integer overflow here.
  val defaultMaxNumExecutorFailures = math.max(3,
    if (effectiveNumExecutors > Int.MaxValue / 2) Int.MaxValue else 2 * effectiveNumExecutors)

  sparkConf.get(MAX_EXECUTOR_FAILURES).getOrElse(defaultMaxNumExecutorFailures)
} 

The result of defaultMaxNumExecutorFailures is calculated first, even if MAX_EXECUTOR_FAILURES is configured now
 
 
 
 
 

Issue Links

• links to: GitHub Pull Request #44691