[SPARK-50288] Executors of stage failed still alive even through stage has been retried

• Type: Question
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0, 3.5.2
• Component/s: Shuffle, Spark Core
• Assignee: Unassigned
• Reporter: Yu-Ting LIN
• Created: 12/Nov/24 07:21
• Updated: 27/Nov/24 11:51

Description

We are executing spark dataframe API with foreachPartition and we observed something that we are not able to explain.

In the figure that we provided, you can see that stage 19 has been retried due to some ShuffleOutputNotFound error and stage 19 has been retried 1.

However, we found that there are still some executors allocated for stage 19 in state RUNNING e.g. partition 371 and 200. In addition, as not yet finished, partition 371 and 200 have been resubmitted again in stage 19 (retry 1). 

Is there any configuration can help us make sure that all executor from the stage failed will be terminated before retry staged been triggered.