[SPARK-12007] Network library's RPC layer requires a lot of copying

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Target Version/s: 1.6.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 26/Nov/15 02:03
• Updated: 26/Feb/16 17:24
• Resolved: 01/Dec/15 01:21

Description

The network library's RPC layer has an external API based on byte arrays, instead of ByteBuffer; that requires a lot of copying since the internals of the library use ByteBuffers (or rather Netty's ByteBuf), and lots of external clients also use ByteBuffer.

The extra copies could be avoided if the API used ByteBuffer instead.

To show an extreme case, look at an RPC send via NettyRpcEnv:

• message is encoded using JavaSerializer, resulting in a ByteBuffer
• the ByteBuffer is copied into a byte array of the right size, since its internal array may be larger than the actual data it holds
• the network library's encoder copies the byte array into a ByteBuf
• finally the data is written to the socket

The intermediate 2 copies could be avoided if the API allowed the original ByteBuffer to be sent instead.

Issue Links

• links to: [Github] Pull Request #9987 (vanzin)