[SPARK-11095] Simplify Netty RPC implementation by using a separate thread pool for each endpoint

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Shixiong Zhu
• Reporter: Reynold Xin
• Created: 14/Oct/15 05:34
• Updated: 21/May/19 18:01
• Resolved: 21/May/19 18:01

Description

The dispatcher class and the inbox class of the current Netty-based RPC implementation is fairly complicated. It uses a single, shared thread pool to execute all the endpoints. This is similar to how Akka does actor message dispatching. The benefit of this design is that this RPC implementation can support a very large number of endpoints, as they are all multiplexed into a single thread pool for execution. The downside is the complexity resulting from synchronization and coordination.

An alternative implementation is to have a separate message queue and thread pool for each endpoint. The dispatcher simply routes the messages to the appropriate message queue, and the threads poll the queue for messages to process.

If the endpoint is single threaded, then the thread pool should contain only a single thread. If the endpoint supports concurrent execution, then the thread pool should contain more threads.

Two additional things we need to be careful with are:

1. An endpoint should only process normal messages after OnStart is called. This can be done by having the thread that starts the endpoint processing OnStart.

2. An endpoint should process OnStop after all normal messages have been processed. I think this can be done by having a busy loop to spin until the size of the message queue is 0.

Issue Links

• is related to: SPARK-5293 Enable Spark user applications to use different versions of Akka