[SPARK-3019] Pluggable block transfer (data plane communication) interface

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 13/Aug/14 23:18
• Updated: 09/Jan/15 08:14
• Resolved: 08/Sep/14 23:01

Description

The attached design doc proposes a standard interface for block transferring, which will make future engineering of this functionality easier, allowing the Spark community to provide alternative implementations.

Block transferring is a critical function in Spark. All of the following depend on it:

• shuffle
• torrent broadcast
• block replication in BlockManager
• remote block reads for tasks scheduled without locality

Issue Links

• is related to: SPARK-3385 Improve shuffle performance
• is related to: SPARK-3059 Spark internal module interface design
• relates to: SPARK-2468 Netty-based block server / client module
• links to: [Github] Pull Request #2240 (rxin)

Sub-Tasks

• Remove Netty specific code in BlockManager: Resolved
• Create LengthBoundedInputStream: Closed