Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.4.1
    • Fix Version/s: 5.6.0
    • Component/s: Broker
    • Labels:
      None

      Description

      One of our installations have several consumers. These consumers subscribe for messages from a queue linked to a virtual topic. All consumers supply a selector. Some consumers connect, process any persisted messages, then disconnect. These connect/disconnect cycles are repeated a few times a day.

      What we've seen is that messages build-up for consumers. These messages does not match the supplied selector. The side effect of this was that we ran into a situation whereby message "got stuck". Increasing the maxPageSize property helped. This is unfortunately a short term solution.

      A simple test was constructed whereby selectorAware was set to true:

      <virtualDestinations>
        <virtualTopic name="VirtualTopic.>" prefix="Consumer.*." selectorAware="true"/>
      </virtualDestinations>
      

      What we noticed is that:

      1. Messages are correctly received by a connected consumer
      2. A consumer that connects, disconnects and re-connects later will loose any messages that were send in the time period it was disconnected.

      This behaviour was unexpected. From the AMQ documention (http://activemq.apache.org/virtual-destinations.html):

      From version 5.4, dispatch from virtual topics to subscription queues can be selectorAware such that only messages that match one of the existing subscribers are actually dispatched. Using this option prevents the build up of unmatched messages when selectors are used by exclusive consumers

      Note: it does not state that the consumer needs to be connected for this feature to work.

      Given the test it looks like subscriptions itself are not persisted, thus the AMQ broker has no idea that it should enqueue a message for a particular subscription queue.

      Would it be possible to add either of:

      1. Persist subscription detail, specifically for the case where the subscription's selector should be applied to the subscription queue
      2. Propagate selectors and the attached subscription queue to the top-level virtual topic so that only interested messages can be delivered to the intended recipient?

      Anything else we can try, supply or help with?

      1. AMQ-3004.patch
        13 kB
        Roelof Naude

        Activity

        Hide
        Rob Davies added a comment -

        Submitted patch - and other bits a pieces -
        Can use the persistent cache as a broker plugin -
        e.g.

        <plugins>
        <virtualSelectorCacheBrokerPlugin persistFile ="selectorcache.data"/>
        </plugins>

        Show
        Rob Davies added a comment - Submitted patch - and other bits a pieces - Can use the persistent cache as a broker plugin - e.g. <plugins> <virtualSelectorCacheBrokerPlugin persistFile ="selectorcache.data"/> </plugins>
        Hide
        Timothy Bish added a comment -

        Would be great if we could get a JUnit test case for this to help determine what the best fix is.

        Show
        Timothy Bish added a comment - Would be great if we could get a JUnit test case for this to help determine what the best fix is.
        Hide
        Gary Tully added a comment -

        @Takar
        Roelof's patch looks very good and is mostly unobtrusive.
        For your use case though I think it only kicks in if all subscriptions are disconnected, which may not be what you want.

        It also needs a junit test case to validate and protect it.
        Maybe you can provide the required test case and some update that will make it work for you?

        Show
        Gary Tully added a comment - @Takar Roelof's patch looks very good and is mostly unobtrusive. For your use case though I think it only kicks in if all subscriptions are disconnected, which may not be what you want. It also needs a junit test case to validate and protect it. Maybe you can provide the required test case and some update that will make it work for you?
        Hide
        Gary Tully added a comment -

        lets try and get to this for 5.6

        Show
        Gary Tully added a comment - lets try and get to this for 5.6
        Hide
        Tkar Akai added a comment -

        We would very much like to get this patch into the trunk!

        Virtual Topics are a perfect match for our use case, we have a single selector per queue (multiple consumers but each using the same selector on a given queue), and need to guarantee that the (filtered) messages are delivered to each queue even when all the consumers disconnect (in our case not intentionally, just dues to some failure or networking issue) and it's not acceptable to miss messages that were published during a disconnect.

        Any chance that this solution or a comparable one will make it to an an official release?

        Show
        Tkar Akai added a comment - We would very much like to get this patch into the trunk! Virtual Topics are a perfect match for our use case, we have a single selector per queue (multiple consumers but each using the same selector on a given queue), and need to guarantee that the (filtered) messages are delivered to each queue even when all the consumers disconnect (in our case not intentionally, just dues to some failure or networking issue) and it's not acceptable to miss messages that were published during a disconnect. Any chance that this solution or a comparable one will make it to an an official release?
        Hide
        Roelof Naude added a comment -

        The attached patch solves the issue for the most part. Consumers need to connect at least once for the selector to be cached. The cached selector will be persisted to the configured file every 10min. It is re-loaded during startup.

        <broker>
        .
        .
        .
           <plugins>
              <spring:ref local="subQueueSelectorCacheBroker"/>
            </plugins>
        .
        .
        .
        </broker>
        
        <spring:bean id="subQueueSelectorCacheBroker" class="org.apache.activemq.plugin.SubQueueSelectorCacheBrokerPlugin">
            <spring:property name="persistFile" value="file:${jboss.server.data.dir}/activemq/selectorCache.dat"/>
         </spring:bean>
        
        Show
        Roelof Naude added a comment - The attached patch solves the issue for the most part. Consumers need to connect at least once for the selector to be cached. The cached selector will be persisted to the configured file every 10min. It is re-loaded during startup. <broker> . . . <plugins> <spring:ref local= "subQueueSelectorCacheBroker" /> </plugins> . . . </broker> <spring:bean id= "subQueueSelectorCacheBroker" class= "org.apache.activemq.plugin.SubQueueSelectorCacheBrokerPlugin" > <spring:property name= "persistFile" value= "file:${jboss.server.data.dir}/activemq/selectorCache.dat" /> </spring:bean>

          People

          • Assignee:
            Rob Davies
            Reporter:
            Roelof Naude
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development