[SPARK-3135] Avoid memory copy in TorrentBroadcast serialization

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Labels: starter
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 19/Aug/14 20:28
• Updated: 18/Oct/14 20:20
• Resolved: 02/Sep/14 03:32

Description

TorrentBroadcast.blockifyObject uses a ByteArrayOutputStream to serialize broadcast object into a single giant byte array, and then separates it into smaller chunks. We should implement a new OutputStream that writes serialized bytes directly into chunks of byte arrays so we don't need the extra memory copy.

Issue Links

• links to: [Github] Pull Request #2054 (rxin)