[SPARK-17574] Cache ShuffleExchange RDD when the exchange is reused

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 17/Sep/16 09:38
• Updated: 17/Sep/16 11:44
• Resolved: 17/Sep/16 11:44

Description

We have the rule ReuseExchange to reuse exchange in the physical plan. However, for a ShuffleExchange we still need to retrieve remote blocks again when the shuffle exchange is reused. We can cache the RDD of the reused ShuffleExchange to avoid transferring remote blocks again.

Issue Links

• links to: [Github] Pull Request #15128 (viirya)