[SPARK-29771] Limit executor max failures before failing the application

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Jackey Lee
• Created: 06/Nov/19 10:10
• Updated: 29/Apr/21 16:21

Description

ExecutorPodsAllocator does not limit the number of executor errors or deletions, which may cause executor restart continuously without application failure.
A simple example for this, add --conf spark.executor.extraJavaOptions=-Xmse after spark-submit, which can make executor restart thousands of times without application failure.

Issue Links

• links to: GitHub Pull Request #26433