[SPARK-27868] Better document shuffle / RPC listen backlog

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.3
• Fix Version/s: 3.0.0
• Component/s: Documentation, Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 28/May/19 22:41
• Updated: 08/Jun/20 17:09
• Resolved: 29/May/19 21:59

Description

The option to control the listen socket backlog for RPC and shuffle servers is not documented in our public docs.

The only piece of documentation is in a Java class, and even that documentation is incorrect:

  /** Requested maximum length of the queue of incoming connections. Default -1 for no backlog. */
  public int backLog() { return conf.getInt(SPARK_NETWORK_IO_BACKLOG_KEY, -1); }

The default value actual causes the default value from the JRE to be used, which is 50 according to the docs.

Issue Links

• links to: GitHub Pull Request #24732
• links to: GitHub Pull Request #27230