[SPARK-7271] Redesign shuffle interface for binary processing

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Later
• Component/s: Shuffle, Spark Core
• Epic Link: Tungsten Phase 2
• Assignee: Josh Rosen
• Reporter: Reynold Xin
• Created: 30/Apr/15 04:40
• Updated: 16/Dec/15 07:46
• Resolved: 16/Dec/15 07:46

Description

Current shuffle interface is not exactly ideal for binary processing.

Sub-Tasks

• Remove hash shuffle file consolidation: Resolved
• Rename HashShufflereader to BlockStoreShuffleReader: Resolved
• Remove ability to set spark.shuffle.spill=false and spark.sql.planner.externalSort=false: Resolved
• Consolidate SortShuffleManager and UnsafeShuffleManager: Resolved
• Refactor DiskBlockObjectWriter to not require BlockId: Resolved