[SPARK-2468] Netty-based block server / client module

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 14/Jul/14 06:39
• Updated: 26/Nov/14 09:46
• Resolved: 05/Nov/14 08:27

Description

Right now shuffle send goes through the block manager. This is inefficient because it requires loading a block from disk into a kernel buffer, then into a user space buffer, and then back to a kernel send buffer before it reaches the NIC. It does multiple copies of the data and context switching between kernel/user. It also creates unnecessary buffer in the JVM that increases GC

Instead, we should use FileChannel.transferTo, which handles this in the kernel space with zero-copy. See http://www.ibm.com/developerworks/library/j-zerocopy/

One potential solution is to use Netty. Spark already has a Netty based network module implemented (org.apache.spark.network.netty). However, it lacks some functionality and is turned off by default.

Issue Links

• is related to: SPARK-3019 Pluggable block transfer (data plane communication) interface
• is related to: SPARK-3385 Improve shuffle performance
• links to: [Github] Pull Request #1907 (rxin)
• links to: [Github] Pull Request #1971 (rxin)