[SPARK-24296] Support replicating blocks larger than 2 GB

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Block Manager, Spark Core
• Assignee: Imran Rashid
• Reporter: Imran Rashid
• Created: 16/May/18 18:22
• Updated: 24/Oct/18 20:34
• Resolved: 21/Aug/18 18:27

Description

Replicating blocks send the entire block data in one frame. This results in a failure on the receiving end for blocks larger than 2GB.

We should change block replication to send the block data as a stream when the block is large (building on the network changes in SPARK-6237). This can use the conf spark.maxRemoteBlockSizeFetchToMem to decided when to replicate as a stream, the same as we do for fetching shuffle blocks and fetching remote RDD blocks.

Issue Links

• causes: SPARK-25827 Replicating a block > 2gb with encryption fails
• links to: [Github] Pull Request #21451 (squito)