[SPARK-47678] Got fetch failed exception when new executor reused same ip address from a previously killed executor

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 3.5.1
• Component/s: Shuffle
• Labels: pull-request-available

Environment

This only happens on Kubernetes, where same ip address can be re-used for new executor pod.

• Assignee: Unassigned
• Reporter: BoYang
• Created: 02/Apr/24 00:40
• Updated: 15/Apr/24 22:46

Description

This is an edge case which caused Spark on Kubernetes getting fetch failed exception when new executor reused same ip address from a previously killed executor.

The new executor checks shuffle block ip address and compares it with its own host address. If the two ip addresses are the same, the new executor will assume the block on its own local disk and try to read it locally. This causes failure since the block is actually on the previously killed executor which happened to have same ip address.

Issue Links

• links to: GitHub Pull Request #45803