Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8
    • Fix Version/s: 0.9
    • Component/s: C++ Broker
    • Labels:
      None

      Description

        1. Problem

      Steadily increasing queue depth is a common symptom of problems in a
      messaging application. The queue may run out of capacity or the broker
      may run out of resources if this condition is not
      rectified. Generating alerts when depth reaches preconfigured
      thresholds would allow administrators to be notified allowing them to
      take corrective action or at least be aware of the situation before it
      becomes more critical.

        1. Solution

      Allow a policy to be specified via the arguments field to queue
      declare, that sets up particular threshold in either message count or
      cumulative message content size, and specifies the level of alert to
      generate. QMF already contains a mechanism for raising alerts. It
      should also be possible to rely on the log output for these alerts if
      log-scraping is used.

      The policy should also allow the frequency at which alerts will be
      regenerated if the depth remains above the configured threshold.

        1. Rationale

      Already use that mechanism to configure queue limits. Have QMF alert
      capabilities. Seems logical to extend the policies to allow earlier
      warning when approaching capacity.

        1. Implementation Notes

      There will be no new components added, this will be added to the code
      around qpid::broker::Queue and qpid::broker::QueuePolicy. (These do
      need refactored somewhat as they are close to unmainatainable at
      present).

      Will need to ensure that existing queue configuration tools
      (i.e. qpid-config) allow configuration of these limits.

        1. Consequences

      Need to ensure that as new features get added around queueing, the
      code is refactored to keep it maitainable and easy to read. Release
      artefacts are unaffected. The feature will need to be documented;
      configuration described in general above details will be added here as
      the implementation progresses. Will not affect backwards
      compatibility.

        Activity

        Gordon Sim created issue -
        Gordon Sim made changes -
        Field Original Value New Value
        Description ## Problem

        Steadily increasing queue depth is a common symptom of problems in a
        messaging application. The queue may run out of capacity or the broker
        may run out of resources if this condition is not
        rectified. Generating alerts when depth reaches preconfigured
        thresholds would allow administrators to be notified allowing them to
        take corrective action or at least be aware of the situation before it
        becomes more critical.

        ## Solution

        Allow a policy to be specified via the arguments field to queue
        declare, that sets up particular threshold in either message count or
        cumulative message content size, and specifies the level of alert to
        generate. QMF already contains a mechanism for raising alerts. It
        should also be possible to rely on the log output for these alerts if
        log-scraping is used.

        The policy should also allow the frequency at which alerts will be
        regenerated if the depth remains above the configured threshold.

        ## Rationale

        Already use that mechanism to configure queue limits. Have QMF alert
        capabilities. Seems logical to extend the policies to allow earlier
        warning when approaching capacity.

        ## Implementation Notes

        There will be no new components added, this will be added to the code
        around qpid::broker::Queue and qpid::broker::QueuePolicy. (These do
        need refactored somewhat as they are close to unmainatainable at
        present).

        Will need to ensure that existing queue configuration tools
        (i.e. qpid-config) allow configuration of these limits.

        ## Consequences

        Need to ensure that as new features get added around queueing, the
        code is refactored to keep it maitainable and easy to read. Release
        artefacts are unaffected. The feature will need to be documented;
        configuration described in general above details will be added here as
        the implementation progresses. Will not affect backwards
        compatibility.
        Hide
        Gordon Sim added a comment -

        Initial patch ready for review at https://reviews.apache.org/r/420/

        Show
        Gordon Sim added a comment - Initial patch ready for review at https://reviews.apache.org/r/420/
        Gordon Sim made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Gordon Sim added a comment -

        For an example/test, see http://svn.apache.org/viewvc/qpid/trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py?view=markup

        There are two new queue arguments - qpid.alert_count and qpid.alert_size - which can be used to control the point at which an alert is generated. An alert is generated when a message is added to the queue and the size or count is at or above the configured alert limit. By default an alert will be generated when reaching 80% of any configured queue limit.

        To avoid excess traffic, once an alert has been raised a further event will not be reissued for a queue for a configurable period. This can be controlled via another queue argument - qpid.alert_repeat_gap - which specifies the minimum interval in seconds between events. The default value for this is one minute (60 seconds).

        To comply with the argument naming already supported in the java broker the following aliases for these arguments are also recognised: x-qpid-maximum-message-count, x-qpid-maximum-message-size, x-qpid-minimum-alert-repeat-gap. (These are a bit more confusing for the c++ broker as qpid.max_size and qpid.max_count are already defined).

        Show
        Gordon Sim added a comment - For an example/test, see http://svn.apache.org/viewvc/qpid/trunk/qpid/tests/src/py/qpid_tests/broker_0_10/threshold.py?view=markup There are two new queue arguments - qpid.alert_count and qpid.alert_size - which can be used to control the point at which an alert is generated. An alert is generated when a message is added to the queue and the size or count is at or above the configured alert limit. By default an alert will be generated when reaching 80% of any configured queue limit. To avoid excess traffic, once an alert has been raised a further event will not be reissued for a queue for a configurable period. This can be controlled via another queue argument - qpid.alert_repeat_gap - which specifies the minimum interval in seconds between events. The default value for this is one minute (60 seconds). To comply with the argument naming already supported in the java broker the following aliases for these arguments are also recognised: x-qpid-maximum-message-count, x-qpid-maximum-message-size, x-qpid-minimum-alert-repeat-gap. (These are a bit more confusing for the c++ broker as qpid.max_size and qpid.max_count are already defined).
        Justin Ross made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Gordon Sim
            Reporter:
            Gordon Sim
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development