ActiveMQ
  1. ActiveMQ
  2. AMQ-2904 failover: protocol doesn't work within a static:() URI in a networkConnector
  3. AMQ-2579

Failover connection recovery needs a new command to indicate recovery completion that can gate dispatch on a recovered consumer

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.3.0
    • Fix Version/s: 5.4.0
    • Component/s: Broker
    • Labels:
      None

      Description

      Unconsumed messages at a consumer need to be rolledback on recovery as they can get redispatched in arbitrary order. see - https://issues.apache.org/activemq/browse/AMQ-2573
      As operations are in progress, like a send transaction, the rollback cannot happen till the send transaction commit completes so it must be async with the failover interruption. Dispatch needs to be gated on completion of the outstanding operations as it currently is with the resolution to AMQ-2573

      However there is the possibility that the broker starts to dispatch to that consumer/connection before recovery is complete and can block the receipt of messages, the response to the send commit for example as the dispatch is waiting for the send to complete so that any unconsumed messages are rolledback in advance of dispatch. With asyncDispatch=false and optimizedDispatch it is possible to simulate this.

      The solution requires two wireformat changes, An indication on a connection that it is recovering (this can be propagated to a consumer) and an indication that recovery is complete such that dispatch on a recovered consumer can complete. An additional AckMode AckRecoveryComplete could do it.
      Thus dispatch would be gated such that it cannot interfere with outstanding work that needs to be restored and completed inorder to correctly clear unconsumed and delivered messages.

        Activity

        Hide
        Gary Tully added a comment -

        Test case for this issue

        Show
        Gary Tully added a comment - Test case for this issue
        Hide
        Gary Tully added a comment -

        noodling around I found consumerControl. The connection state tracker can recover consumers with a prefetch of 0, such that no message will be eagerly dispatched and once client side recovery processing is complete, send a consumerControl to reset the prefetch values to the original state.
        In this way, the client connections are responsible for recovery (rather than flagging a connection as recovered and letting it handle consumers for the duration of recovery).
        Recovery on the client side is complete when all consumers have cleared their unconsumed and delivered messages which is dependent on resumption of the connection and responses to outstanding sends or acks or transaction commits. Only the ActiveMQConneciton knows when this is complete so it needs to callback to the failover transport when all is done.
        To make this work, ConsumerControl needs an destination such that it can be used to locate a subscription on the broker side. Previously it has only been used to send info from the broker to a client connection. This requires a wireformat update.

        Show
        Gary Tully added a comment - noodling around I found consumerControl. The connection state tracker can recover consumers with a prefetch of 0, such that no message will be eagerly dispatched and once client side recovery processing is complete, send a consumerControl to reset the prefetch values to the original state. In this way, the client connections are responsible for recovery (rather than flagging a connection as recovered and letting it handle consumers for the duration of recovery). Recovery on the client side is complete when all consumers have cleared their unconsumed and delivered messages which is dependent on resumption of the connection and responses to outstanding sends or acks or transaction commits. Only the ActiveMQConneciton knows when this is complete so it needs to callback to the failover transport when all is done. To make this work, ConsumerControl needs an destination such that it can be used to locate a subscription on the broker side. Previously it has only been used to send info from the broker to a client connection. This requires a wireformat update.
        Hide
        Gary Tully added a comment -

        resolved in r902908
        introduces wire format update to version 6

        Show
        Gary Tully added a comment - resolved in r902908 introduces wire format update to version 6

          People

          • Assignee:
            Gary Tully
            Reporter:
            Gary Tully
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development