[SPARK-10997] Netty-based RPC env should support a "client-only" mode.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 08/Oct/15 03:03
• Updated: 04/Mar/20 18:04
• Resolved: 02/Nov/15 18:27

Description

The new netty RPC still behaves too much like akka; it requires both client (e.g. an executor) and server (e.g. the driver) to listen for incoming connections.

That is not necessary, since sockets are full-duplex and RPCs should be able to flow either way on any connection. Also, because the semantics of the netty-based RPC don't exactly match akka, you get weird issues like SPARK-10987.

Supporting a client-only mode also reduces the number of ports Spark apps need to use.

Issue Links

• is related to: SPARK-20605 Deprecate not used AM and executor port configuration
• links to: [Github] Pull Request #9210 (vanzin)