[SPARK-28239] Allow TCP connections created by shuffle service auto close on YARN NodeManagers

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Shuffle, Spark Core, YARN

Environment

Hadoop2.6.0-CDH5.8.3(netty3)
Spark2.4.0(netty4)

Configs:
spark.shuffle.service.enabled=true

• Assignee: Unassigned
• Reporter: Deegue
• Created: 03/Jul/19 07:30
• Updated: 17/May/20 18:13

Description

When executing shuffle tasks, TCP connections(on port 7337 by default) will be established by shuffle service.
It will like:

However, some of the TCP connections are still busy when the task is actually finished. These connections won't close automatically until we restart the NodeManager process.

Connections pile up and NodeManagers are getting slower and slower.

These unclosed TCP connections stay busy and it seem doesn't take effect when I set ChannelOption.SO_KEEPALIVE to true according to SPARK-23182.

So the solution is setting ChannelOption.AUTO_CLOSE to true, and after which our cluster(running 10000+ jobs / day) is processing normally.

Issue Links

• links to: GitHub Pull Request #25057