[SPARK-6521] Bypass network shuffle read if both endpoints are local

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Assignee: Unassigned
• Reporter: xukun
• Created: 25/Mar/15 03:04
• Updated: 09/Aug/19 23:50
• Resolved: 01/Dec/15 08:57

Description

In the past, executor read other executor's shuffle file in the same node by net. This pr make that executors in the same node read local shuffle file In sort-based Shuffle. It will reduce net transport.

Issue Links

• is related to: SPARK-27651 Avoid the network when block manager fetches shuffle blocks from the same host
• links to: [Github] Pull Request #5178 (viper-kun)
• links to: [Github] Pull Request #9478 (maropu)