ActiveMQ
  1. ActiveMQ
  2. AMQ-3488

Temporary destinations' DestinationInfo commands over VM transport prevent connection closure

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.5.0
    • Fix Version/s: 5.6.0
    • Component/s: Transport
    • Labels:
      None
    • Regression:
      Regression

      Description

      ActiveMQTempDestinations hold onto the source connection that created them. When the source connection is closed, its temporary destinations are deleted. Unfortunately, DestinationInfo commands convey the destination object with its source connection to all other connections on the same VMTransportServer. Hence when these other connections are closed, they attempt to delete the source connection's temporary destination.

      Note that enabling marshaling on the VM transport works around this bug because destination serialization and deserialization does not maintain the source connection reference.

      This bug was not present in 5.4.2.

        Activity

        Hide
        Dan Retzlaff added a comment -

        The attached test case demonstrates this bug by having a consumer active on the source connection, which causes a "Consumer is consuming from the temporary destination" exception when a separate connection is created and closed.

        Show
        Dan Retzlaff added a comment - The attached test case demonstrates this bug by having a consumer active on the source connection, which causes a "Consumer is consuming from the temporary destination" exception when a separate connection is created and closed.
        Hide
        Timothy Bish added a comment -

        Fixed in trunk, thanks for the great test case.

        Show
        Timothy Bish added a comment - Fixed in trunk, thanks for the great test case.
        Hide
        David Johle added a comment -

        I have some additional notes about some other side effects I've encountered that I can attribute to this same bug. Just posting them here in case anyone else not running 5.6.0 yet is running into them and trying to figure out what is going on.

        I noticed that, when using a temporary destination and then closing a connection and subsequently receiving the "Consumer is consuming..." exception, the consumer for the queue I closed a connection to was left dangling.

        That is, I start up my application with X amount of consumers and look at the queue via JMX under org.apache.activemq->myBrokerName->Queue->myQueuName->Attributes->Subscriptions
        and see X amount subscriptions to that queue as expected. I then instruct my application to shut down Y amount of consumers, and the Subscriptions count remains at X instead of the expected value of X - Y = Z.

        If I disable the portion of my application that uses the temporary queue, then no exceptions happen, and the above shutdown test does cause the Subscriptions count to drop to Z accordingly.

        What's worse is that it's not just excess JMX ObjectNames left behind, but the actual consumer subscriptions they relate to are also still there and registered with ActiveMQ / bound to that queue as MessageListeners. That can be verified via JMX as well, under org.apache.activemq->myBrokerName->Subscription->Non-Durable->Queue->myQueueName.

        So what happens next is...

        I then trigger my application to produce a bunch of messages into myQueueName. Most of them are consumed by the remaining Z consumers, but a certain number of them just get stuck in the queue. And not just any number, but precisely the number Y * prefetchSize.

        If I restart my entire application the messages then get re-delivered to the fresh consumers and flushed out of myQueueName. That is, unless I don't get to it in time, at which point they expire and end up in the DLQ instead.

        Again, disabling the use of temporary qeueus avoids this issue. The problem, of course, is that I need the temps for proper app functionality!

        I also tested this scenario with the current 5.6.0-SNAPSHOT and can confirm that it does indeed resolve the issue.

        Show
        David Johle added a comment - I have some additional notes about some other side effects I've encountered that I can attribute to this same bug. Just posting them here in case anyone else not running 5.6.0 yet is running into them and trying to figure out what is going on. I noticed that, when using a temporary destination and then closing a connection and subsequently receiving the "Consumer is consuming..." exception, the consumer for the queue I closed a connection to was left dangling. That is, I start up my application with X amount of consumers and look at the queue via JMX under org.apache.activemq->myBrokerName->Queue->myQueuName->Attributes->Subscriptions and see X amount subscriptions to that queue as expected. I then instruct my application to shut down Y amount of consumers, and the Subscriptions count remains at X instead of the expected value of X - Y = Z. If I disable the portion of my application that uses the temporary queue, then no exceptions happen, and the above shutdown test does cause the Subscriptions count to drop to Z accordingly. What's worse is that it's not just excess JMX ObjectNames left behind, but the actual consumer subscriptions they relate to are also still there and registered with ActiveMQ / bound to that queue as MessageListeners. That can be verified via JMX as well, under org.apache.activemq->myBrokerName->Subscription->Non-Durable->Queue->myQueueName. So what happens next is... I then trigger my application to produce a bunch of messages into myQueueName. Most of them are consumed by the remaining Z consumers, but a certain number of them just get stuck in the queue. And not just any number, but precisely the number Y * prefetchSize. If I restart my entire application the messages then get re-delivered to the fresh consumers and flushed out of myQueueName. That is, unless I don't get to it in time, at which point they expire and end up in the DLQ instead. Again, disabling the use of temporary qeueus avoids this issue. The problem, of course, is that I need the temps for proper app functionality! I also tested this scenario with the current 5.6.0-SNAPSHOT and can confirm that it does indeed resolve the issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Dan Retzlaff
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development