[SPARK-3119] Re-implement TorrentBroadcast

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark Core
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 19/Aug/14 06:28
• Updated: 20/Aug/14 05:11
• Resolved: 20/Aug/14 05:11

Description

TorrentBroadcast is unnecessarily complicated:

1. It tracks a lot of mutable states, such as total number of bytes, number of blocks fetched.
2. It has at least two data structures that are not needed: TorrentInfo and TorrentBlock.
3. It uses getSingle on executors to get the block instead of getLocal, resulting in an extra roundtrip to look up the location of the block when the block doesn't exist yet.
4. It has a metadata block that is completely unnecessary.
5. It does an extra memory copy during deserialization to copy all the blocks into a single giant array.

Issue Links

• is related to: SPARK-3115 Improve task broadcast latency for small tasks
• links to: [Github] Pull Request #2030 (rxin)