[SPARK-11235] Support streaming data using network library

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 21/Oct/15 18:27
• Updated: 11/Jan/19 04:11
• Resolved: 04/Nov/15 17:12

Description

As part of the work to implement SPARK-11140, it would be nice to have the network library efficiently stream data over a connection.

Currently all it has is the shuffle data protocol, which is not very efficient for large files; it requires the whole file to be buffered on the receiver side before the receiver can do anything. For large files, that comes at a huge cost in memory. You can chunk large files but that requires the client to ask for each chunk separately.

Instead, a similar approach but allowing the data to be processed as it arrives would be a lot more efficient, and make it easier to implement the file server in the referenced bug.

Issue Links

• blocks: SPARK-11140 Replace file server in driver with RPC-based alternative
• links to: [Github] Pull Request #9206 (vanzin)
• links to: GitHub Pull Request #9206