[DIRMINA-678] NioProcessor 100% CPU usage on Linux (epoll selector bug)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-M4
• Fix Version/s: 2.0.3
• Component/s: Core
• Environment: CentOS 5.x, 32/64-bit, 32/64-bit Sun JDK 1.6.0_12, also _11/_10/_09 and Sun JDK 1.7.0 b50, Kernel 2.6.18-92.1.22.el5 and also older versions,
• Assignee: Unassigned
• Reporter: Serge Baranov
• Created: 19/Mar/09 00:41
• Updated: 17/May/19 14:21
• Resolved: 09/Apr/11 16:30

Description

It's the same bug as described at http://jira.codehaus.org/browse/JETTY-937 , but affecting MINA in the very similar way.

NioProcessor threads start to eat 100% resources per CPU. After 10-30 minutes of running depending on the load (sometimes after several hours) one of the NioProcessor starts to consume all the available CPU resources probably spinning in the epoll select loop. Later, more threads can be affected by the same issue, thus 100% loading all the available CPU cores.

Sample trace:

NioProcessor-10 [RUNNABLE] CPU time: 5:15
sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int)
sun.nio.ch.EPollArrayWrapper.poll(long)
sun.nio.ch.EPollSelectorImpl.doSelect(long)
sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
sun.nio.ch.SelectorImpl.select(long)
org.apache.mina.transport.socket.nio.NioProcessor.select(long)
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run()
org.apache.mina.util.NamePreservingRunnable.run()
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()

It seems to affect any NIO based Java server applications running in the specified environment.
Some projects provide workarounds for similar JDK bugs, probably MINA can also think about a workaround.

As far as I know, there are at least 3 users who experience this issue with Jetty and all of them are running CentOS (some distribution default setting is a trigger?). As for MINA, I'm not aware of similar reports yet.

Issue Links

• is related to: DIRMINA-1111 100% CPU (epoll bug) on 2.1.x, Linux only
• relates to: ZOOKEEPER-427 ZooKeeper server unexpectedly high CPU utilisation
• relates to: DIRMINA-681 Strange CPU peak occuring at fixed interval when several thousand connections active
• requires: DIRMINA-886 isBrokenConnection