[SPARK-4740] Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Zhang, Liye
• Created: 04/Dec/14 07:57
• Updated: 08/Jul/19 17:24
• Resolved: 15/Dec/14 18:18

Description

When testing current spark master (1.3.0-snapshot) with spark-perf (sort-by-key, aggregate-by-key, etc), Netty based shuffle transferService takes much longer time than NIO based shuffle transferService. The network throughput of Netty is only about half of that of NIO.

We tested with standalone mode, and the data set we used for test is 20 billion records, and the total size is about 400GB. Spark-perf test is Running on a 4 node cluster with 10G NIC, 48 cpu cores per node and each executor memory is 64GB. The reduce tasks number is set to 1000.

—
Reynold update on Dec 15, 2014: The problem is that in NIO we have multiple connections between two nodes, but in Netty we only had one. We introduced a new config option spark.shuffle.io.numConnectionsPerPeer to allow users to explicitly increase the number of connections between two nodes. SPARK-4853 is a follow-up ticket to investigate setting this automatically by Spark.

Issue Links

• causes: SPARK-28261 Flaky test: org.apache.spark.network.TransportClientFactorySuite.reuseClientsUpToConfigVariable
• is related to: SPARK-4853 Automatically adjust the number of connections between two peers to achieve good performance
• links to: [Github] Pull Request #3625 (rxin)
• links to: [Github] Pull Request #3667 (rxin)