Duplicate entries in flush-list, consequent unnecessary system-calls and overzealous wakeups and calls to write(2)

• Key: DIRMINA-65
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Trustin Lee
• Reporter: Mark Atwell
• Affects Version/s: 0.7.0, 0.7.1, 0.7.2
• Fix Version/s: 0.7.3

Description

The following shows up under heavy load - writing numerous small messages across a slow(ish) network and/or to a slow peer i.e. when a writer starts to see back-log and writes don't necessarily complete.

In SocketIoProcessor:

- scheduleFlush() blindly adds the session to the queue even if it is already present. This means that when the selector thread wakes up it may make numerous calls to (relatively slow) write on the same session.

- flushSession() calls scheduleFlush() and always wakes up the selector thread. This means that flushSessions() and flush() is called for each application write - even when the socket may already be backed up with queued traffic. This is exacerbated by the problem above.

- scheduleFlush() is called from flushSession() which is called from (at least) SocketSessionManagerFilterChain for every application write.

I 'hacked' a test fix which modified scheduleFlush() to only add the session if it wasn't already in the queue and returns a boolean indicating whether the session was added. flushSession() checks this return value and only wakes the selector thread if the session has been added (i.e. it isn't already backed up). In this way writes are not unnecessarily triggered.

This worked though it also highlighted (or brought into greater focus? :o) an apparent timing problem.

java.io.IOException: Invalid argument
at sun.nio.ch.DevPollArrayWrapper.registerMultiple(Native Method)
at sun.nio.ch.DevPollArrayWrapper.updateRegistrations(DevPollArrayWrapper.java:209)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:157)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.mina.io.socket.SocketIoProcessor$Worker.run(SocketIoProcessor.java:562)

This isn't fatal though as the retry logic means that the next time 'round the state - presumably of the selection keys - has been brought into alignemnt?

Comments

[Mark Atwell]

My tentative fix. It has had limited testing. Still has the original SVN ID line in it and some of the whitespace may not be to your taste!

[Trustin Lee]

Thank you for posting a patch for this issue first of all, Mark. I appologize for this late reply. I'm pretty busy these days. :)

Did you get any clue about 'java.io.IOException: Invalid argument' exception message, BTW? If the selector recovers itself automatically in the next loop, it shouldn't be problem. WDYT?

[Mark Atwell]

No problem, we've all got other things to do! :o)

I haven't investigated the IOException any further. I guess the choice comes down to:

- Whether there is a simple, correct, low-overhead solution - I haven't had time to think about this yet. i.e. what the actual problem is (though I think I'm probably about right), and whether the fix is 'all of the above'.

- The overhead of avoiding an event that, for me at least, so far has been infrequent.

- The risk of not reporting it and accidentally hiding/losing a different more significant problem.

BTW I've logged a bug with Sun about being able to do a ByteBuffer(et al).release().

[Trustin Lee]

I checked in the fix. I found it is much simpler and efficient to add the checking code to SessionManagerFilterChain implementations.

Could you please try again with the new build and close this issue after testing?