[SPARK-47702] Shuffle service endpoint is not removed from the locations list when RDD block is removed form a node.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0, 3.5.4
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Attila Zsolt Piros
• Reporter: mahesh kumar behera
• Created: 03/Apr/24 03:33
• Updated: 23/Oct/24 17:48
• Resolved: 04/Oct/24 15:36

Description

If SHUFFLE_SERVICE_FETCH_RDD_ENABLED is set to true, driver stores both executor end point and the external shuffle end points for a RDD block. When the RDD is migrated, the location info is updated to add the end point corresponds to new location and the old end point is removed. But currently, only the executor end point is removed. The shuffle service end point is not removed. This cause failure during RDD read if the shuffle service end point is chosen due to task locality.

Issue Links

• links to: GitHub Pull Request #45836
• links to: GitHub Pull Request #47779