Qpid
  1. Qpid
  2. QPID-4728

Allow option to configure credit on Federation sessions.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23
    • Fix Version/s: 0.23
    • Component/s: C++ Broker
    • Labels:
      None

      Description

      By default, sessions created for Federation links use unlimited credit. This JIRA requests the ability to configure the credit used by a Federation link. E.g.:

      qpid-route --credit 100 route add ....

        Activity

        Hide
        Fraser Adams added a comment -

        If this relates to what I think it does (I've never been able to articulate the problem very well!!) then this sounds extremely useful for increasing the reliability of federation links.

        I've had an issue for ages where there "appeared" to be a memory leak on some of our source brokers, but after a lot of head scratching it turned out to be an issue due to there being "unbounded buffers" on the federation links.

        In my scenario we run a topology whereby a real time producer writes to a co-located broker and that broker is federated to another broker etc.

        In most cases this works very well but in one particular set of nodes in our topology the producer is particularly high rate and quite "bursty" and what we'd see is the memory utilisation of the source broker start to increase noticeably and eventually we'd hit swap.

        What ultimately appeared to be causing the issue was that the network between the brokers was "throttling" the communication (I did say it was a fast bursty producer) so from the perspective of the source broker we appeared to have a fast producer with a slow consumer.

        As I was trying to get to the bottom of this I ended up trying to reproduce it at the lowest level I could, so wrote simple qpid::messaging and qpid::client producer consumer clients to try and "emulate" the federation bridge code and found that the qpid::client code appeared to "leak" but the "equivalent" qpid::messaging code didn't - then it dawned on me that by default qpid::messaging has consumer capacity of 1 (so I generally set it to something that gives reasonable prefetch) whereas qpid::client defaults to unlimited.

        Our eventual workaround was to have this particular fast bursty producer deliver directly to the second stage broker rather than to its co-located broker, the theory being that by doing this the rate limiting effect of the network actually becomes our friend, so rather than the fast data source hitting a broker pretty much as fast as it can it now hits it as fast as the network supports, which then generally avoids the unbounded buffer growing in an unmanageable way.

        It's a pretty subtle issue that took us ages to diagnose and caused more than a few sleepless nights, so I'm pretty much +1 million on adding the ability to make the federation link capacity bounded.

        Show
        Fraser Adams added a comment - If this relates to what I think it does (I've never been able to articulate the problem very well!!) then this sounds extremely useful for increasing the reliability of federation links. I've had an issue for ages where there "appeared" to be a memory leak on some of our source brokers, but after a lot of head scratching it turned out to be an issue due to there being "unbounded buffers" on the federation links. In my scenario we run a topology whereby a real time producer writes to a co-located broker and that broker is federated to another broker etc. In most cases this works very well but in one particular set of nodes in our topology the producer is particularly high rate and quite "bursty" and what we'd see is the memory utilisation of the source broker start to increase noticeably and eventually we'd hit swap. What ultimately appeared to be causing the issue was that the network between the brokers was "throttling" the communication (I did say it was a fast bursty producer) so from the perspective of the source broker we appeared to have a fast producer with a slow consumer. As I was trying to get to the bottom of this I ended up trying to reproduce it at the lowest level I could, so wrote simple qpid::messaging and qpid::client producer consumer clients to try and "emulate" the federation bridge code and found that the qpid::client code appeared to "leak" but the "equivalent" qpid::messaging code didn't - then it dawned on me that by default qpid::messaging has consumer capacity of 1 (so I generally set it to something that gives reasonable prefetch) whereas qpid::client defaults to unlimited. Our eventual workaround was to have this particular fast bursty producer deliver directly to the second stage broker rather than to its co-located broker, the theory being that by doing this the rate limiting effect of the network actually becomes our friend, so rather than the fast data source hitting a broker pretty much as fast as it can it now hits it as fast as the network supports, which then generally avoids the unbounded buffer growing in an unmanageable way. It's a pretty subtle issue that took us ages to diagnose and caused more than a few sleepless nights, so I'm pretty much +1 million on adding the ability to make the federation link capacity bounded.
        Hide
        Ken Giusti added a comment -

        Proposed fix available at Reviewboard:

        https://reviews.apache.org/r/10373/

        Show
        Ken Giusti added a comment - Proposed fix available at Reviewboard: https://reviews.apache.org/r/10373/
        Show
        Ken Giusti added a comment - Submission: http://svn.apache.org/viewvc?view=revision&revision=1467107
        Hide
        Justin Ross added a comment -
        Show
        Justin Ross added a comment - Released in Qpid 0.24, http://qpid.apache.org/releases/qpid-0.24/index.html

          People

          • Assignee:
            Ken Giusti
            Reporter:
            Ken Giusti
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development