Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)

• Key: DIRMINA-65
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Trustin Lee
• Reporter: Mark Atwell
• Affects Version/s: 0.7.0, 0.7.1, 0.7.2
• Fix Version/s: 0.7.3

Description

The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.

In SocketIoProcessor:

- scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.

- flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.

- scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.

I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.

This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.

java.io.IOException: Invalid argument
at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)

This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?