[SPARK-2956] Support transferring large blocks in Netty network module

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.1.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 10/Aug/14 23:42
• Updated: 15/Aug/14 02:09
• Resolved: 15/Aug/14 02:09

Description

The existing Netty shuffle implementation does not support large blocks.

The culprit is in FileClientHandler.channelRead0().

We should add a LengthFieldBasedFrameDecoder to the pipeline.

Issue Links

• links to: Github pull request