[SPARK-45937] Fix documentation of spark.executor.maxNumFailures

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Thomas Graves
• Created: 15/Nov/23 14:59
• Updated: 20/Nov/23 15:56
• Resolved: 20/Nov/23 15:56

Description

https://issues.apache.org/jira/browse/SPARK-41210 added support for spark.executor.maxNumFailures on Kubernetes, it made this config generic and deprecated the yarn version.  This config isn't documented and defaults are not documented.

 

https://github.com/apache/spark/commit/40872e9a094f8459b0b6f626937ced48a8d98efb

\

It also added spark.executor.failuresValidityInterval.

 

Both need to have default values specified for yarn and k8s, it also needs to remove the yarn documentation for equivalent configs spark.yarn.max.executor.failures configuration

 
 
 

Issue Links

• duplicates: SPARK-45969 Document configuration change of executor failure tracker
• is broken by: SPARK-41210 Window based executor failure tracking mechanism