Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
When users start to create large topologies the storm netty messaging layer
uses lots of threads. This has resulted in OOMs because the default ulimit on most linux distros is around 4000 processes. It looks like the messaging layer wants to have one thread per server it is connected to, so that means the total number of other workers in the System.
For one particular case we saw.
1 (Curator delay thread)
1 (Curator Event Processor)
1 (Finalizer)
1 (GC???)
1 (Storm messaging recv thread asking netty for messages)
1 (Thread pool polling on a Synchronous queue???)
1 (ZK Connection)
1 (ZK epoll)
2 (???)
2 (Netty epoll)
6 (Timer Thread)
15 (Disruptor consume batches)
104 (Netty Thread pool taking messages to be sent)
and this process was dieing with OOMs because it could not create any more netty threads.
Looking at the code it appears that come from two different things. First The Client code is using it's own thread pool for each Client instead of sharing a thread pool, but also the protocol itself blocks the thread in takeMessages() if there are no messages to send.
So we need to make the thread pool shared between all of the clients and modify the protocol so that takeMessages does not block. But with it not blocking we also need a way to have Client.send write directly to the Channel in some situations so that the messages still are sent.