[SPARK-16593] Provide a pre-fetch mechanism to accelerate shuffle stage.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: Spark Core
• Labels: bulk-closed features
• Assignee: Unassigned
• Reporter: Lewin Ma
• Created: 17/Jul/16 14:28
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

Currently, the `NettyBlockRpcServer` will reading data through BlockManager, while the block is not cached in memory, the data should be read from DISK first, then into MEM. I wonder if we implement a mechanism add a message contains the blockIds that the same as the openBlock message but one loop ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer to the reduce side.

Issue Links

• links to: [Github] Pull Request #14239 (f7753)