[SPARK-37029] Modify the assignment logic of dirFetchRequests variables

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.1.2, 3.2.0
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: jinhai
• Created: 17/Oct/21 09:45
• Updated: 20/Oct/21 06:17
• Resolved: 20/Oct/21 06:17

Description

In the ShuffleBlockFetcherIterator.fetchHostLocalBlocks method, we generate dirFetchRequests based on externalShuffleServiceEnabled. But in fact, the MapStatus object generated in the shuffle write phase had already generated the BlockManagerId object according to externalShuffleServiceEnabled in the BlockManager.initialize method.

So we don't need to judge it again.

Issue Links

• links to: [Github] Pull Request #34304 (manbuyun)