Qpid
  1. Qpid
  2. QPID-4591

mechanism to detect when messages are overwritten in ring-type queues

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18
    • Fix Version/s: 0.23
    • Component/s: C++ Broker
    • Labels:
      None

      Description

      A way to determine when a ring queue is full and old messages are being deleted. Also need a way to determine when the ring queue is no longer full.

      1. bz691411.patch1
        5 kB
        Ernest Allen

        Activity

        Hide
        Ernest Allen added a comment -

        A suggested implementation is:
        Send the deleted messages to the alternate exchange associated with a queue. That would let any application detect them by binding to that exchange as appropriate. That is also more susceptible to ACL control.

        To avoid this mechanism causing build up of messages that the ring policy is attempting to avoid, the queue bound to the alternate exchange could have be a ring queue of size 1, meaning it only ever held the last message which I think would be sufficient to notify of the dropping of messages.

        So for the application, the following would happen:

        • create an alternate exchange for use with the ring queue
        • create the ring queue, associating it with the alternate exchange
        • create an overflow ring queue with max-count=1
        • bind the overflow queue with the alternate exchange
        • watch the overflow queue. When a message appears, the original queue is full and has just overflowed.
        • remove the overflow message
        Show
        Ernest Allen added a comment - A suggested implementation is: Send the deleted messages to the alternate exchange associated with a queue. That would let any application detect them by binding to that exchange as appropriate. That is also more susceptible to ACL control. To avoid this mechanism causing build up of messages that the ring policy is attempting to avoid, the queue bound to the alternate exchange could have be a ring queue of size 1, meaning it only ever held the last message which I think would be sufficient to notify of the dropping of messages. So for the application, the following would happen: create an alternate exchange for use with the ring queue create the ring queue, associating it with the alternate exchange create an overflow ring queue with max-count=1 bind the overflow queue with the alternate exchange watch the overflow queue. When a message appears, the original queue is full and has just overflowed. remove the overflow message
        Hide
        Jimmy Jones added a comment -

        I think QMF should be able to help you

        Show
        Jimmy Jones added a comment - I think QMF should be able to help you
        Hide
        Jakub Scholz added a comment -

        ad QMF)
        An QMF event informing about the overwriting isn't a bad idea. But the problem with QMF is that it cannot be properly secured with ACL. So for some Qpid users (in our situation a broker used to connect hundreds of customers to our service) we cannot really provide the access to the QMF without causing possible security issues.

        ad alt exchange)
        I can imagine to create a construct using an alternate exchange as described by Ernest. But such construct will be not exactly user friendly. This will get additionally complicated when you are working with multiple ring-type queues receiving the same messages at the same time - some of them having the messages overwritten and some of them not. You would need to have for each ring type queue a separate alternate exchange with another small ring-type signal queue. Also, it is a bit "overkill" to resend all the messages to alternate exchange just to give a "signal" that they are being overwritten. When you create the ring type queue, you usually expect the messages to get lost - you just want to know when it happens.

        I would maybe suggest following solutions ...

        1) Assigning a new custom header to the remaining message(s) which will mark that some messages were overwritten (i.e. something similar to the redelivered flag).

        or

        2) Adding a queue level message sequencing which would work in a similar way as the exchange level sequencing, but the sequence IDs will be assigned only to the messages which are routed into the queue (+ the sequence will be persisted to possibly survive the restart of the broker - unlike with the exchange level sequencing). The sequence IDs can be used by the client to detect gaps. (this is from my perspective the best solution)

        Show
        Jakub Scholz added a comment - ad QMF) An QMF event informing about the overwriting isn't a bad idea. But the problem with QMF is that it cannot be properly secured with ACL. So for some Qpid users (in our situation a broker used to connect hundreds of customers to our service) we cannot really provide the access to the QMF without causing possible security issues. ad alt exchange) I can imagine to create a construct using an alternate exchange as described by Ernest. But such construct will be not exactly user friendly. This will get additionally complicated when you are working with multiple ring-type queues receiving the same messages at the same time - some of them having the messages overwritten and some of them not. You would need to have for each ring type queue a separate alternate exchange with another small ring-type signal queue. Also, it is a bit "overkill" to resend all the messages to alternate exchange just to give a "signal" that they are being overwritten. When you create the ring type queue, you usually expect the messages to get lost - you just want to know when it happens. I would maybe suggest following solutions ... 1) Assigning a new custom header to the remaining message(s) which will mark that some messages were overwritten (i.e. something similar to the redelivered flag). or 2) Adding a queue level message sequencing which would work in a similar way as the exchange level sequencing, but the sequence IDs will be assigned only to the messages which are routed into the queue (+ the sequence will be persisted to possibly survive the restart of the broker - unlike with the exchange level sequencing). The sequence IDs can be used by the client to detect gaps. (this is from my perspective the best solution)
        Hide
        Justin Ross added a comment -

        An approach currently under review:

        https://reviews.apache.org/r/11009/

        Show
        Justin Ross added a comment - An approach currently under review: https://reviews.apache.org/r/11009/
        Hide
        Ernest Allen added a comment -

        Adds mechanism to store and retrieve a queue level sequence number in message properties.

        To enable storing of sequence numbers, declare queue with argument qpid.queue_msg_sequence set to user defined key.
        To retrieve the sequence number from a message, get the message properties using the same user defined key.

        std::string addr("my-queue;"
        " {create:always, delete:always,"
        " node: {type: queue, x-declare: {arguments:"
        " {qpid.queue_msg_sequence:my_seq_key, qpid.policy_type:ring, qpid.max_count:4}}}}");

        Sender sender = session.createSender(addr);

        ...

        Receiver receiver = session.createReceiver("my-queue");

        Message response = receiver.fetch();
        uint32_t seqNo = response.getProperties()["my_seq_key"];
        if (seqNo - lastSeqNo > 1)
        //Gap detected. Messages were overwritten
        ...

        Show
        Ernest Allen added a comment - Adds mechanism to store and retrieve a queue level sequence number in message properties. To enable storing of sequence numbers, declare queue with argument qpid.queue_msg_sequence set to user defined key. To retrieve the sequence number from a message, get the message properties using the same user defined key. std::string addr("my-queue;" " {create:always, delete:always," " node: {type: queue, x-declare: {arguments:" " {qpid.queue_msg_sequence:my_seq_key, qpid.policy_type:ring, qpid.max_count:4}}}}"); Sender sender = session.createSender(addr); ... Receiver receiver = session.createReceiver("my-queue"); Message response = receiver.fetch(); uint32_t seqNo = response.getProperties() ["my_seq_key"] ; if (seqNo - lastSeqNo > 1) //Gap detected. Messages were overwritten ...
        Hide
        Gordon Sim added a comment -

        Ernie's patch applied as http://svn.apache.org/r1485001

        Show
        Gordon Sim added a comment - Ernie's patch applied as http://svn.apache.org/r1485001
        Hide
        Robbie Gemmell added a comment -

        New python tests excluded from runs against the Java broker in http://svn.apache.org/r1485953

        Show
        Robbie Gemmell added a comment - New python tests excluded from runs against the Java broker in http://svn.apache.org/r1485953
        Hide
        Gordon Sim added a comment -

        (oops, sorry Robbie!)

        Show
        Gordon Sim added a comment - (oops, sorry Robbie!)
        Hide
        Robbie Gemmell added a comment -

        Not to worry, only noticed this morning that they were failing as we had previously broken the test job itself

        Show
        Robbie Gemmell added a comment - Not to worry, only noticed this morning that they were failing as we had previously broken the test job itself
        Hide
        Justin Ross added a comment -

        Gordon, is there any more work remaining for this issue?

        Show
        Justin Ross added a comment - Gordon, is there any more work remaining for this issue?
        Hide
        Gordon Sim added a comment -

        Not directly. As noted in the review linked to above, the addAnnotations() method this change uses currently has a significant impact on performance. I have put up a patch for review at https://reviews.apache.org/r/11329/ that minimises the impact.

        Show
        Gordon Sim added a comment - Not directly. As noted in the review linked to above, the addAnnotations() method this change uses currently has a significant impact on performance. I have put up a patch for review at https://reviews.apache.org/r/11329/ that minimises the impact.
        Hide
        Justin Ross added a comment -
        Show
        Justin Ross added a comment - Released in Qpid 0.24, http://qpid.apache.org/releases/qpid-0.24/index.html

          People

          • Assignee:
            Gordon Sim
            Reporter:
            Ernest Allen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development