[SPARK-4853] Automatically adjust the number of connections between two peers to achieve good performance

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 15/Dec/14 18:16
• Updated: 17/May/20 18:31
• Resolved: 21/May/19 05:36

Description

As discovered in SPARK-4740, performance of the new Netty transport can be impacted by the total number of active connections. This manifests itself when the following 3 conditions are true:
(1) # spinning disks per node is large (doesn't affect SSDs)
(2) # cores/node is large
(3) # nodes is small

In 1.2, we created a new config variable spark.shuffle.io.numConnectionsPerPeer that allows users to explicitly increase the number of connections between any two nodes. Ideally, we should have Spark automatically figure out the optimal (or near optimal) setting is so users don't have to worry about this config option.

Issue Links

• relates to: SPARK-4740 Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey