[SPARK-36784] Handle DNS failures in Executor without causing FetchFailed

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.0
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: Thejdeep Gudivada
• Created: 16/Sep/21 17:57
• Updated: 06/May/22 10:11

Description

DNS failures caused on the executor nodes causing shuffle nodes to be added to the exclude list. We should handle this failure and not cause a FetchFailed exception in such cases.

This helps in two things : 

1. Stage won't be resubmitted due to FetchFailedException
2. Spark's exclude listing also won't exclude the shuffle service node when the problem is
indeed with the current executor host.

Issue Links

• links to: [Github] Pull Request #34024 (thejdeep)
• links to: [Github] Pull Request #34024 (thejdeep)