We have been seeing a large number of SASL authentication (RPC requests) timing out with the external shuffle service.
The issue and all the analysis we did is described here:
I added a LoggingHandler to netty pipeline and realized that even the channel registration is delayed by 30 seconds.
In the Spark External Shuffle service, the boss event group and the worker event group are same which is causing this delay.
When the load at the shuffle service increases, since the worker threads are busy with existing channels, registering new channels gets delayed.
The fix is simple. I created a dedicated boss thread event loop group with 1 thread.
This fixed the issue.
We just need 1 thread in the boss group because there is only a single server bootstrap.