|
[
Permlink
| « Hide
]
Martin Ritchie added a comment - 02/Nov/06 12:13 PM
Multi Threaded SocketIOProcessor with test app
Martin Ritchie made changes - 02/Nov/06 12:13 PM
Updated version to work with Mina HEAD and only use backport and slf4j.
Martin Ritchie made changes - 03/Nov/06 03:37 PM
I tried the Multi Threaded SocketIOProcessor over the weekend and I come across some issues.
Environment: - Mina-Head, using JUC instead of backport-util-concurrent and the ReadWriteThreadModel from QPid Project - Linux CentOS 4.4 - Dual Xeon 3GHz, 2GB RAM - JDK 6 VM (build 103) - Server and Message Producer on the same Machine, Message Consumer on a remote machine (both machines are identical in all aspects) - Message size is ~3KB. Every Message has a length header and XML payload (Soap). - I use un-pooled heap ByteBuffers. The test is simple, every 5 seconds the Message Producer opens a thread/socket to the server, sends a batch of 25000 messages and closes the socket when it's done, on occasion there is some overlap in that a thread starts before the previous one finishes. On the consumer side there is some basic accounting to check if every message of the batch is received. There is a problem in the MultiThreadSocketIoProcessor.read(), it enters a endless loop after a client closes the connection, I "fixed it" bringing the outer "try/catch" block out from the "for" loop. I may have introduced some bugs because the performance of this Multi Threaded SocketIOProcessor is much worse than the default, it goes down from 6500 msg/sec to 50 msg/sec. I can't complete the sending of a single batch of messages without errors of this kind: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55) at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:69) at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:271) at org.apache.mina.transport.socket.nio.MultiThreadSocketIoProcessor.processRead(MultiThreadSocketIoProcessor.java:368) at org.apache.mina.transport.socket.nio.MultiThreadSocketIoProcessor.access$15(MultiThreadSocketIoProcessor.java:359) at org.apache.mina.transport.socket.nio.MultiThreadSocketIoProcessor$ReadWorker.run(MultiThreadSocketIoProcessor.java:902) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43) at java.lang.Thread.run(Thread.java:619) Is there anything that you guys want me to do to try to diagnose what's going wrong? Regards -- Luis Neves If we perform I/O in more than one threads, it is inevitable to get CancelledKeyException because one thread can't detect if the channel is already closed and the selector key is cancelled when the other thread closed the channel and cancelled the key. We can synchronized it, but the cost is very huge. We can simply ignore CancelledKeyException, but there's a risk here because we might not be able to distinguish the expected exception and the unexpected exception (bug).
synchronized -> synchronize :)
Anyway, I'm just curious why using two threads can boost performance when read and write operations are just memory copy between kernel buffer and Java heap memory. It could boost performance for smaller number of connections, but not for a number of connections. Please correct if I am missing something. Basically SocketIoProcessor shouldn't be biased to the any specific I/O operation. If it is biased to write operations, I think it's a bug and should be fixed. This boosts performance because reads and writes for a given connection can occur in parallel, which is not something that can happen at present. You can load-balance by using multiple IoProcessors, but still not read+write simultaneously for the same connection.
I believe the CanceledKeyException is fixable. Luis, can you construct a simple failing testcase for it? :) The performance increase comes when the kernel buffer doesn't fill up and the doFlush(Session) exist very rarely has there is an infinite for loop that is excited by a full kernel buffer.
Addition of locking code should be possible, I'll have a look Updated to lock usage of selector keys. This should prevent the CancelledKeyException. I haven't been able to accurately test this as I have never seen a CancelledKeyException, If someone has a test application that can demonstrate this then it would be great if I could use it to solve the problem.
I still get a consistent increase in speed with this over the standard Mina SocketIOProcessor
Martin Ritchie made changes - 08/Nov/06 08:27 AM
I've tested the updated code with this test case:
http://websites.labs.sapo.pt/mina/MinaTestCase.zip I no longer see the CancelledKeyException, however, with the above test case if the server and the message producer are on the same machine the server deadlocks,I think is a deadlock, there is thread dump inside the archive for inspection) This can also be a bug introduced by my local changes, I'm not running a stock Mina. The changes: - changed the ThreadPoolExecutor used in org.apache.qpid.pool.ReferenceCountingExecutorService, but I see the same behaviour with the default ExecutorService. - replaced the use of backport-util-concurrent with JUC. - changed the org.apache.mina.filter.ReadThrottleFilterBuilder.getThreadPoolFilterEntryName() in order to use the org.apache.qpid.pool.ReadWriteThreadModel. - commented out the logging in the the org.apache.qpid.pool.Event constructor. Regards -- Luis Neves I have had a look at the stack traces but I don't see any deadlock? Which threads to you think are deadlocked?
Also could you try this with an unmodified MINA so that we can rule out your modifications? We will also try running your tests. You are right there is no deadlocked, I miss spoked, the behaviour I'm seeing is that the server stops reading. Sorry for using the wrong terminology.
With the default Mina I don't see that behaviour. -- Luis Neves OK, it is very useful to know that it does work with default Mina. We had verified that the server stops reading in your test case but had not checked the result with unmodified MINA. We will see if we can reproduce that here.
The starvation problem appears to be due to a bug in the ReadThrottleFilterBuilder.
There does seem to be a few issue with the current version The way that the trafficMask is updated in the MTSIOP only the read thread can update the traffic mask I have updated it to allow both threads to call the method. The write timeout was updating the read selection key not the write key. There were a few instances where a CancelledKeyException could occur as not all of the key methods that throw this exception were synchronized. I have made those changes so I'll up load this new version. MultiThreadSocketIoProcessor.java:
Updated synchronization Called doUpdateTrafficMask from both threads MultiThreadSocketSessionImpl.java: Added a getReadSelectionKey()
Martin Ritchie made changes - 15/Nov/06 03:16 PM
I didn't have enough time to look into the code yet due to overload at work, so please forgive my silly question: does MultiThreadSocketIoProcessor limit the number of bytes to be written per session per loop? If not, the fairness improves, but it might not be fair enough actually, because one outstanding session can hinder other ordinary sessions write operation.
It doesn't currently limit the amount of data written per session. It is like the standard IOProcessor and relies on the kernel buffer filling up to stop a session flushing.
It is a good point that to be fair to all sessions you would need to limit the amount written. I currently limit the amount read to 512k per session (would be nice to make this configurable) so doing something similar for writes would restore the balance. I have made that change locally but don't have a serious test framework to give a quantifiable measure of the benefit. I shall repost the zip with that change. Update to include a 512k maximum write per session. Mirroring the maximum read per session. This should result in improved fairness when multiple sessions are constantly writing large amounts of data.
Martin Ritchie made changes - 16/Nov/06 10:06 AM
I am sorry for getting back to you very late. You know, I don't have such a blazingly fast device. ;)
I've just resolved Status for this issue ?
Unfortuately I have not had time recently to continue this work. The Qpid project currently relies on the ByteBuffer being uncompressed between data reads as reported in
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Improvement
Open
Major
New Multi threaded SocketIOProcessor to improve fairness of socket reads/writes
