As discovered in
SPARK-4740, performance of the new Netty transport can be impacted by the total number of active connections. This manifests itself when the following 3 conditions are true:
(1) # spinning disks per node is large (doesn't affect SSDs)
(2) # cores/node is large
(3) # nodes is small
In 1.2, we created a new config variable spark.shuffle.io.numConnectionsPerPeer that allows users to explicitly increase the number of connections between any two nodes. Ideally, we should have Spark automatically figure out the optimal (or near optimal) setting is so users don't have to worry about this config option.