Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.0.0
    • Fix Version/s: 5.1.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      Linux

      Description

      For some time now there have been various bug reports about ActiveMQ "blocking", "not receiving messages", "running into a deadlock" etc. Since I encoutered such deadlocks now and then, too, I eventually wrote up a JUnit testing scenario for this stuff. I found out that deadlocks can be quite easily reproduced. The symptoms are that the producer thread is sending or committing while the consumer thread is receiving or committing - and none of them can advance. One of the threads is always stuck in a blocking queue.

      Here's a sample output of my testing class:
      An ActiveMQ deadlock has been discovered. The following threads seem to be involved:

      Thread "producer" is inactive since 16 seconds after 358719 status changes. The current status is COMMITTING
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
      java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
      java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
      org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
      org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:76)
      org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1172)
      org.apache.activemq.TransactionContext.commit(TransactionContext.java:259)
      org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:494)
      de.rainer_klute.activemq.ProducerThread.run(ProducerThread.java:162)

      Thread "consumer" is inactive since 16 seconds after 1807 status changes. The current status is RECEIVING
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:485)
      org.apache.activemq.MessageDispatchChannel.dequeue(MessageDispatchChannel.java:75)
      org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:404)
      org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:452)
      org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:504)
      de.rainer_klute.activemq.ConsumerThread.run(ConsumerThread.java:183)

      The following factors seem to increase the probability of a deadlock:

      • small values for memoryUsage
      • working transacted in the consumer (not always necessary but "helps")
      • many messages in the persistence store (to be achieved via a long delay before the consumer starts to read messages)
      1. AMQ-1490_result-002.txt
        11 kB
        Rainer Klute
      2. AMQ-1490_result-001.txt
        85 kB
        Rainer Klute
      3. AMQ-1490_memory-001.png
        54 kB
        Rainer Klute
      4. ActiveMQ_Testcases.jar
        38 kB
        Rainer Klute
      5. ActiveMQ_Testcases.jar
        49 kB
        Rainer Klute

        Activity

        Hide
        Rainer Klute added a comment -

        JUnit test cases for this issue. See file index.html or AMQ-1490.html for details.

        Show
        Rainer Klute added a comment - JUnit test cases for this issue. See file index.html or AMQ-1490 .html for details.
        Hide
        Rainer Klute added a comment -

        I just checked this issue against ActiveMQ 5.0.0 RC3 and found it still valid. And I got additional warnings like these:

        [junit] INFO: ActiveMQ Message Broker (localhost, ID:xxxxxxxx-17170-1194934336239-0:0) is shutting down
        [junit] 13.11.2007 07:13:03 org.apache.activemq.ActiveMQConnection onAsyncException
        [junit] WARNUNG: Async exception with no exception listener: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed.
        [junit] org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed.
        [junit] at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:200)
        [junit] at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:118)
        [junit] at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:42)
        [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        [junit] at java.lang.Thread.run(Thread.java:619)
        [junit] 13.11.2007 07:13:03 org.apache.activemq.AdvisoryConsumer dispose
        [junit] INFO: Failed to send remove command: javax.jms.JMSException: Peer (vm://localhost#1) disposed.
        [junit] javax.jms.JMSException: Peer (vm://localhost#1) disposed.
        [junit] at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
        [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1154)
        [junit] at org.apache.activemq.AdvisoryConsumer.dispose(AdvisoryConsumer.java:56)
        [junit] at org.apache.activemq.ActiveMQConnection.cleanup(ActiveMQConnection.java:1326)
        [junit] at org.apache.activemq.ActiveMQConnection.transportFailed(ActiveMQConnection.java:2027)
        [junit] at org.apache.activemq.ActiveMQConnection$4.run(ActiveMQConnection.java:1663)
        [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        [junit] at java.lang.Thread.run(Thread.java:619)
        [junit] Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed.
        [junit] at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:87)
        [junit] at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        [junit] at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:59)
        [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1152)
        [junit] ... 7 more
        [junit] 13.11.2007 07:13:03 org.apache.activemq.ActiveMQConnection transportFailed
        [junit] WARNUNG: Cleanup failed
        [junit] javax.jms.JMSException: Peer (vm://localhost#1) disposed.
        [junit] at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
        [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1154)
        [junit] at org.apache.activemq.TransactionContext.begin(TransactionContext.java:200)
        [junit] at org.apache.activemq.ActiveMQSession.doStartTransaction(ActiveMQSession.java:1653)
        [junit] at org.apache.activemq.ActiveMQMessageConsumer.acknowledge(ActiveMQMessageConsumer.java:823)
        [junit] at org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:651)
        [junit] at org.apache.activemq.ActiveMQSession.dispose(ActiveMQSession.java:575)
        [junit] at org.apache.activemq.ActiveMQConnection.cleanup(ActiveMQConnection.java:1332)
        [junit] at org.apache.activemq.ActiveMQConnection.transportFailed(ActiveMQConnection.java:2027)
        [junit] at org.apache.activemq.ActiveMQConnection$4.run(ActiveMQConnection.java:1663)
        [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        [junit] at java.lang.Thread.run(Thread.java:619)
        [junit] Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed.
        [junit] at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:87)
        [junit] at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        [junit] at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:59)
        [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1152)
        [junit] ... 11 more

        Show
        Rainer Klute added a comment - I just checked this issue against ActiveMQ 5.0.0 RC3 and found it still valid. And I got additional warnings like these: [junit] INFO: ActiveMQ Message Broker (localhost, ID:xxxxxxxx-17170-1194934336239-0:0) is shutting down [junit] 13.11.2007 07:13:03 org.apache.activemq.ActiveMQConnection onAsyncException [junit] WARNUNG: Async exception with no exception listener: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed. [junit] org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed. [junit] at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:200) [junit] at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:118) [junit] at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:42) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) [junit] at java.lang.Thread.run(Thread.java:619) [junit] 13.11.2007 07:13:03 org.apache.activemq.AdvisoryConsumer dispose [junit] INFO: Failed to send remove command: javax.jms.JMSException: Peer (vm://localhost#1) disposed. [junit] javax.jms.JMSException: Peer (vm://localhost#1) disposed. [junit] at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62) [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1154) [junit] at org.apache.activemq.AdvisoryConsumer.dispose(AdvisoryConsumer.java:56) [junit] at org.apache.activemq.ActiveMQConnection.cleanup(ActiveMQConnection.java:1326) [junit] at org.apache.activemq.ActiveMQConnection.transportFailed(ActiveMQConnection.java:2027) [junit] at org.apache.activemq.ActiveMQConnection$4.run(ActiveMQConnection.java:1663) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) [junit] at java.lang.Thread.run(Thread.java:619) [junit] Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed. [junit] at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:87) [junit] at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) [junit] at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:59) [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1152) [junit] ... 7 more [junit] 13.11.2007 07:13:03 org.apache.activemq.ActiveMQConnection transportFailed [junit] WARNUNG: Cleanup failed [junit] javax.jms.JMSException: Peer (vm://localhost#1) disposed. [junit] at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62) [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1154) [junit] at org.apache.activemq.TransactionContext.begin(TransactionContext.java:200) [junit] at org.apache.activemq.ActiveMQSession.doStartTransaction(ActiveMQSession.java:1653) [junit] at org.apache.activemq.ActiveMQMessageConsumer.acknowledge(ActiveMQMessageConsumer.java:823) [junit] at org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:651) [junit] at org.apache.activemq.ActiveMQSession.dispose(ActiveMQSession.java:575) [junit] at org.apache.activemq.ActiveMQConnection.cleanup(ActiveMQConnection.java:1332) [junit] at org.apache.activemq.ActiveMQConnection.transportFailed(ActiveMQConnection.java:2027) [junit] at org.apache.activemq.ActiveMQConnection$4.run(ActiveMQConnection.java:1663) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) [junit] at java.lang.Thread.run(Thread.java:619) [junit] Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://localhost#1) disposed. [junit] at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:87) [junit] at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) [junit] at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:59) [junit] at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1152) [junit] ... 11 more
        Hide
        Hiram Chirino added a comment -

        I reviewed you test case and noticed that your consumer uses the same connection as the producer. It is possible to get into a deadlock situation when this occurs. If your producer blocks due to it waiting space on the broker, it will lock up the connection. At that point the consumer will not be able to consume anymore messages either.

        I believe your deadlocks will go away if you use separate connections for the consumer and producer threads.

        Show
        Hiram Chirino added a comment - I reviewed you test case and noticed that your consumer uses the same connection as the producer. It is possible to get into a deadlock situation when this occurs. If your producer blocks due to it waiting space on the broker, it will lock up the connection. At that point the consumer will not be able to consume anymore messages either. I believe your deadlocks will go away if you use separate connections for the consumer and producer threads.
        Hide
        Rainer Klute added a comment -

        Using the same connection for the producer as well as for the consumer is neither uncommon nor is it contrary to the JMS specification. I'd say since the JMS specification allows it, ActiveMQ should support it on principle.

        Some more rationale: With separate connections you would have to diffentiate between "producing" and "consuming" connections. This would be unnecessarily complex - not to mention the many connections' resource consumption. An application's JMS setup would be too static. It should be possible to build up complex communication scenarios all over the same connection, and it should even be possible to change the communicators' roles (producer or consumer) over time.

        I'd be grateful if you could address this issue.

        What can I do until then? In my project's scenario I have a few producers (at present only one) writing messages to a topic and about 200 consumers reading them based on message properties. Should I really use different connections for all of them or are two connections sufficient?

        Show
        Rainer Klute added a comment - Using the same connection for the producer as well as for the consumer is neither uncommon nor is it contrary to the JMS specification. I'd say since the JMS specification allows it, ActiveMQ should support it on principle. Some more rationale: With separate connections you would have to diffentiate between "producing" and "consuming" connections. This would be unnecessarily complex - not to mention the many connections' resource consumption. An application's JMS setup would be too static. It should be possible to build up complex communication scenarios all over the same connection, and it should even be possible to change the communicators' roles (producer or consumer) over time. I'd be grateful if you could address this issue. What can I do until then? In my project's scenario I have a few producers (at present only one) writing messages to a topic and about 200 consumers reading them based on message properties. Should I really use different connections for all of them or are two connections sufficient?
        Hide
        Albert Strasheim added a comment -

        This is starting to sound like an issue I reported some time back, namely AMQ-1148, which hasn't received much attention in recent months.

        There is nothing in the JMS specification to indicate that one should use separate connections for producers and consumers and doing so can have various implications for the design of an application, so I would agree with Rainer Klute that this is something that should probably be looked at carefully.

        Show
        Albert Strasheim added a comment - This is starting to sound like an issue I reported some time back, namely AMQ-1148 , which hasn't received much attention in recent months. There is nothing in the JMS specification to indicate that one should use separate connections for producers and consumers and doing so can have various implications for the design of an application, so I would agree with Rainer Klute that this is something that should probably be looked at carefully.
        Hide
        Rainer Klute added a comment -

        Meanwhile I am pretty sure that the "use two connections" approach does not work. My application has two producers and a few dozens of consumers, and now - desperate as I am - each session has its own connection or, to put it the other way round, each connection has only a single session. Nevertheless I still end up with a lot of threads hanging around like shown in the original bug description (see above).

        Show
        Rainer Klute added a comment - Meanwhile I am pretty sure that the "use two connections" approach does not work. My application has two producers and a few dozens of consumers, and now - desperate as I am - each session has its own connection or, to put it the other way round, each connection has only a single session. Nevertheless I still end up with a lot of threads hanging around like shown in the original bug description (see above).
        Hide
        Rainer Klute added a comment -

        I did some further investigation and changed by JUnit test to use a configurable number of producers and consumers. The good news is that it demonstrates a bug with a very small amount of messages already. I one of the test scenarios I have 2 producers. Each one sends 2 messages to a topic. The single consumer should receive 4 messages. However, this fails depending on the combination of whether the producers or the consumers or both operate transacted or not. Increase the number of messages and other combinations fail. (I'll check in the JUnit test soon.)

        Show
        Rainer Klute added a comment - I did some further investigation and changed by JUnit test to use a configurable number of producers and consumers. The good news is that it demonstrates a bug with a very small amount of messages already. I one of the test scenarios I have 2 producers. Each one sends 2 messages to a topic. The single consumer should receive 4 messages. However, this fails depending on the combination of whether the producers or the consumers or both operate transacted or not. Increase the number of messages and other combinations fail. (I'll check in the JUnit test soon.)
        Hide
        Rainer Klute added a comment -

        Modified JUnit tests as mentioned in the comment above. See index.html for details!

        Show
        Rainer Klute added a comment - Modified JUnit tests as mentioned in the comment above. See index.html for details!
        Hide
        Rainer Klute added a comment - - edited

        Small fix in the JUnit test.

        If running the test with 2 messages (i.e. 1 per producer) the test fails differently from when running with many messages (e.g. 20,000 per producer).

        Show
        Rainer Klute added a comment - - edited Small fix in the JUnit test. If running the test with 2 messages (i.e. 1 per producer) the test fails differently from when running with many messages (e.g. 20,000 per producer).
        Hide
        Rainer Klute added a comment -

        Just tried the stuff with a TCP transport instead of a VM transport: No substantial change except for the places where the deadlocks occur.

        Show
        Rainer Klute added a comment - Just tried the stuff with a TCP transport instead of a VM transport: No substantial change except for the places where the deadlocks occur.
        Hide
        Rob Davies added a comment -

        The consumer(s) need to be initialized before you send messages to them -
        e.g. when creating the ConsumerThreads in the ReliabilityTest - call consumer.init() - which actually creates the JMS consumer.

        ActiveMQ only sends topic messages to consumers that exist - or have existed (if they are durable).
        Also - please don't call inactivity a deadlock

        Show
        Rob Davies added a comment - The consumer(s) need to be initialized before you send messages to them - e.g. when creating the ConsumerThreads in the ReliabilityTest - call consumer.init() - which actually creates the JMS consumer. ActiveMQ only sends topic messages to consumers that exist - or have existed (if they are durable). Also - please don't call inactivity a deadlock
        Hide
        Rainer Klute added a comment - - edited

        Please look once more: I do initialize the consumer threads and create the consumers - see line 282 in ReliabilityTest.java.

        If there were no consumers, they wouldn't receive any messages at all. However, the usually do receive some messages, but not all.

        Show
        Rainer Klute added a comment - - edited Please look once more: I do initialize the consumer threads and create the consumers - see line 282 in ReliabilityTest.java. If there were no consumers, they wouldn't receive any messages at all. However, the usually do receive some messages, but not all.
        Hide
        Rob Davies added a comment -

        your calling init in the wrong place - move it to line 252

        Show
        Rob Davies added a comment - your calling init in the wrong place - move it to line 252
        Hide
        Rainer Klute added a comment -

        Yup, you are right! I now moved initializing and starting the consumers before the producers. However, the behaviour is still buggy, i.e. the two producers hanging in a commit() while the consumer is in a receive(). The corrected JUnit test will follow in a minute.

        Show
        Rainer Klute added a comment - Yup, you are right! I now moved initializing and starting the consumers before the producers. However, the behaviour is still buggy, i.e. the two producers hanging in a commit() while the consumer is in a receive(). The corrected JUnit test will follow in a minute.
        Hide
        Rainer Klute added a comment -

        Fixed JUnit tests

        Show
        Rainer Klute added a comment - Fixed JUnit tests
        Hide
        Rob Davies added a comment -

        Potential fix by SVN revision revision 599129
        Rainer would you mind verifying please ?

        Show
        Rob Davies added a comment - Potential fix by SVN revision revision 599129 Rainer would you mind verifying please ?
        Hide
        Rainer Klute added a comment -

        The good news is: Yes, it works! Great! Thanks, Rob!

        The bad news is: Some test cases take a tremendous amount of heap space now. My former max memory setting of -Xmx256K was insufficient; with -Xmx1024K the test application runs. This gave me cause to investigate that behaviour more closely and I had jconsole watch the memory usage. The result you'll find in attachment AMQ-1490_memory-001.png. I did some drawing on that screenshot to demarcate the test cases. The wall clock times on the X axis you can correlate to the detailed test case protocol in AMQ-1490_result-001.txt.

        The scenario is that two producers send each 100,000 messages to a topic and one consumer reads them all. The differences between them are whether producing and consuming sessions are transactional or not.

        Interesting results:

        • The enormous memory consumption only occurs if the producers operate transactionally and the consumer does not.
        • Test case 0001 (transactional producers) starts to consume much memory. However, after a mark-and-sweep garbage collection things stay moderate.
        • The execution times vary very much: There's a factor of 7.7 between the fastest and the slowest run.
        • The fastests test cases are those without transactional producers.
        Show
        Rainer Klute added a comment - The good news is: Yes, it works! Great! Thanks, Rob! The bad news is: Some test cases take a tremendous amount of heap space now. My former max memory setting of -Xmx256K was insufficient; with -Xmx1024K the test application runs. This gave me cause to investigate that behaviour more closely and I had jconsole watch the memory usage. The result you'll find in attachment AMQ-1490 _memory-001.png. I did some drawing on that screenshot to demarcate the test cases. The wall clock times on the X axis you can correlate to the detailed test case protocol in AMQ-1490 _result-001.txt. The scenario is that two producers send each 100,000 messages to a topic and one consumer reads them all. The differences between them are whether producing and consuming sessions are transactional or not. Interesting results: The enormous memory consumption only occurs if the producers operate transactionally and the consumer does not. Test case 0001 (transactional producers) starts to consume much memory. However, after a mark-and-sweep garbage collection things stay moderate. The execution times vary very much: There's a factor of 7.7 between the fastest and the slowest run. The fastests test cases are those without transactional producers.
        Hide
        Rainer Klute added a comment -

        Memory consumption screenshot (see last comment)

        Show
        Rainer Klute added a comment - Memory consumption screenshot (see last comment)
        Hide
        Rainer Klute added a comment -

        Detailed test results (see last comment)

        Show
        Rainer Klute added a comment - Detailed test results (see last comment)
        Hide
        Rob Davies added a comment -

        There's additional fixes here by SVN revision 600891
        If there isn't enough memory for the broker to finish a transaction the broker will block - so if you see producers pause this is the likely reason

        Show
        Rob Davies added a comment - There's additional fixes here by SVN revision 600891 If there isn't enough memory for the broker to finish a transaction the broker will block - so if you see producers pause this is the likely reason
        Hide
        Rainer Klute added a comment -

        I'd be happy if that would be so. However, instead of pausing the test case still fails with an OutOfMemoryError when I specify -Xmx256M or -Xmx512M.

        With 768 MB it works. The funny thing is that it runs faster with 768 MB than with 1024 MB. Maybe its swapping that disturbs in the latter case.

        Show
        Rainer Klute added a comment - I'd be happy if that would be so. However, instead of pausing the test case still fails with an OutOfMemoryError when I specify -Xmx256M or -Xmx512M. With 768 MB it works. The funny thing is that it runs faster with 768 MB than with 1024 MB. Maybe its swapping that disturbs in the latter case.
        Hide
        Rob Davies added a comment -

        Hi Rainer,
        This has taken a while to track down - but the ack list in amq could grow excessively large. For my tests - the memory profile is good with 100,000 msgs per producer - if you get a chance - please confirm

        using svn revision 602440

        thanks,

        Rob

        Show
        Rob Davies added a comment - Hi Rainer, This has taken a while to track down - but the ack list in amq could grow excessively large. For my tests - the memory profile is good with 100,000 msgs per producer - if you get a chance - please confirm using svn revision 602440 thanks, Rob
        Hide
        Rainer Klute added a comment -

        Hi Rob,
        now there's another problem: When the consumer is not yet receiving messages, the producers now block after sending a bunch of messages. They should spool the messages to disk instead. (My real-world application generates dozens or hundreds of messages per second while there might be now consumer available for days.)

        Show
        Rainer Klute added a comment - Hi Rob, now there's another problem: When the consumer is not yet receiving messages, the producers now block after sending a bunch of messages. They should spool the messages to disk instead. (My real-world application generates dozens or hundreds of messages per second while there might be now consumer available for days.)
        Hide
        Rob Davies added a comment -

        Hi Rainer,
        the problem is the size of the transaction - currently you need to either have smaller transactions - or enough memory to accommodate them,
        As this problem has moved on since the original issue - can we close this and add a new feature that would allow spooling to disk for in-memory transactions?

        thanks,

        Rob

        Show
        Rob Davies added a comment - Hi Rainer, the problem is the size of the transaction - currently you need to either have smaller transactions - or enough memory to accommodate them, As this problem has moved on since the original issue - can we close this and add a new feature that would allow spooling to disk for in-memory transactions? thanks, Rob
        Hide
        Rainer Klute added a comment -

        Rob,
        I am presently running my test cases

        • without the one with the large transaction size but
        • with more messages than before (500,000 per producer, with 2 producers and 1 consumer).

        The flow control issues seems indeed to be solved, so yes, we can close this issue.

        However, with the increase of the number of messages, I get all sorts of errors from the AMQ store, i.e.

        • notes about messages that could not have been entered into the store because of an IndexOutOfBoundException, and
        • notes about messages that could not have been recoved from the store.
          Before creating another issue, I'll update to the very latest ActiveMQ snapshot and retry.
        Show
        Rainer Klute added a comment - Rob, I am presently running my test cases without the one with the large transaction size but with more messages than before (500,000 per producer, with 2 producers and 1 consumer). The flow control issues seems indeed to be solved, so yes, we can close this issue. However, with the increase of the number of messages, I get all sorts of errors from the AMQ store, i.e. notes about messages that could not have been entered into the store because of an IndexOutOfBoundException, and notes about messages that could not have been recoved from the store. Before creating another issue, I'll update to the very latest ActiveMQ snapshot and retry.
        Hide
        Rainer Klute added a comment - - edited

        Oh no! With the latest snapshot (subversion revision 605158) I get blocks/deadlocks again! Producers and consumers each hanging in a commit()!

        Show
        Rainer Klute added a comment - - edited Oh no! With the latest snapshot (subversion revision 605158) I get blocks/deadlocks again! Producers and consumers each hanging in a commit()!
        Hide
        Rob Davies added a comment -

        Could you attach a stack dump of where they hang in a commit ? - I can't reproduce

        Show
        Rob Davies added a comment - Could you attach a stack dump of where they hang in a commit ? - I can't reproduce
        Hide
        Rainer Klute added a comment -

        Hm, when I tried once again to reproduce it, I couldn't. Perhaps my grace period of 15 seconds for deadlock/blocking was too short and the algorithm detected a block that wasn't one. Now it requires 2 minutes without any send/receive activity before claiming that something blocks.

        I'd say let's close this issue. I'll continue to stress test ActiveMQ anyway and will submit a new issue soon!

        Rob, thanks a lot for fixing this issue!

        Show
        Rainer Klute added a comment - Hm, when I tried once again to reproduce it, I couldn't. Perhaps my grace period of 15 seconds for deadlock/blocking was too short and the algorithm detected a block that wasn't one. Now it requires 2 minutes without any send/receive activity before claiming that something blocks. I'd say let's close this issue. I'll continue to stress test ActiveMQ anyway and will submit a new issue soon! Rob, thanks a lot for fixing this issue!
        Hide
        Rob Davies added a comment -

        Phew! Thanks Rainer for you're help!!

        Show
        Rob Davies added a comment - Phew! Thanks Rainer for you're help!!
        Hide
        Rainer Klute added a comment -

        Sorry, but I have to reopen this issue again!

        While the test scenario "multiple producers, single consumer" seems to work fine in most cases, the scenario "one producer, multiple consumers" fails soon and reproducibly with just 2 consumers and as few as 6,000 messages. (In my production environment I'll have about 600 consumers and about 250 messages per second.)

        I shall re-submit a further development of my JUnit test case shortly.

        Show
        Rainer Klute added a comment - Sorry, but I have to reopen this issue again! While the test scenario "multiple producers, single consumer" seems to work fine in most cases, the scenario "one producer, multiple consumers" fails soon and reproducibly with just 2 consumers and as few as 6,000 messages. (In my production environment I'll have about 600 consumers and about 250 messages per second.) I shall re-submit a further development of my JUnit test case shortly.
        Hide
        Rainer Klute added a comment -

        Sample output of the "1 producer, 2 consumers" scenario

        Show
        Rainer Klute added a comment - Sample output of the "1 producer, 2 consumers" scenario
        Hide
        Rainer Klute added a comment -

        The latest version of my test cases.

        Show
        Rainer Klute added a comment - The latest version of my test cases.
        Hide
        Rainer Klute added a comment -

        Seems to work with revision 627336 if transaction size is not too big. However, if neither producer nor consumers are transacted, performance is best.

        Show
        Rainer Klute added a comment - Seems to work with revision 627336 if transaction size is not too big. However, if neither producer nor consumers are transacted, performance is best.
        Hide
        Lari Hotari added a comment -

        I'm using Jencks Inbound/Outbound JMS configuration for consuming & producing messages and quite often there's a 2 minute delay in message delivery after the producer has replied. This happens only in ActiveMQ 5.0. I'll try nightly builds to see if they fix the problem. Maybe JCA/RA has the same issue?

        Show
        Lari Hotari added a comment - I'm using Jencks Inbound/Outbound JMS configuration for consuming & producing messages and quite often there's a 2 minute delay in message delivery after the producer has replied. This happens only in ActiveMQ 5.0. I'll try nightly builds to see if they fix the problem. Maybe JCA/RA has the same issue?

          People

          • Assignee:
            Rob Davies
            Reporter:
            Rainer Klute
          • Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development