ActiveMQ
  1. ActiveMQ
  2. AMQ-816

new transport for load balancing client requests across many brokers

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: NEEDS_REVIEW
    • Component/s: None
    • Labels:
      None

      Description

      Rather than creating store and forward networks, it might be nice to have a kind of composite transport where...

      • consumers are created on each broker found/discovered. This allows messages to be sent to any broker and consumed by any consumer
      • producers could dynmically choose which broker to send a message to (or could just pick one broker per session/producer)

      this allows for a load balancing layer at the client side which avoids the need for store/forward networks (which results in more network traffic and often increases load on the brokers).

      So it basically pushes load back to the clients. The downside of this appoach is that the clients have more connections to brokers - but given the linear scalability of this, it sounds a great idea to me at least

        Activity

        Hide
        Hiram Chirino added a comment -

        Sounds similar to the fanout transport we currently have. Except that in the fanout transport it's backwards.
        We broadcast the publish to multiple brokers and only consume from 1.

        Show
        Hiram Chirino added a comment - Sounds similar to the fanout transport we currently have. Except that in the fanout transport it's backwards. We broadcast the publish to multiple brokers and only consume from 1.
        Hide
        james strachan added a comment -

        These issues are similar, though AMQ-816 focusses on the client side and achieving load balancing across brokers without requiring the brokers to be networked

        Show
        james strachan added a comment - These issues are similar, though AMQ-816 focusses on the client side and achieving load balancing across brokers without requiring the brokers to be networked
        Hide
        james strachan added a comment -

        Was talking about this possible new feature with some users the other day and we tried to come up with a name to refer to this new kinda transport. I was previously struggling to think of one - then we ended up using the term 'jedi'. The name has kinda stuck - I think its cool

        So we could call this the jedi: transport

        Show
        james strachan added a comment - Was talking about this possible new feature with some users the other day and we tried to come up with a name to refer to this new kinda transport. I was previously struggling to think of one - then we ended up using the term 'jedi'. The name has kinda stuck - I think its cool So we could call this the jedi: transport
        Hide
        james strachan added a comment -

        We currently have the fanount transport which does most of this - the main thing to add is the ability to choose the broker to send a command to depending on the context. e.g. when using a transaction, choose a broker and use it for the entire transaction (unless the broker dies). When sending a MessageAck use the broker that sent the original message etc.

        Show
        james strachan added a comment - We currently have the fanount transport which does most of this - the main thing to add is the ability to choose the broker to send a command to depending on the context. e.g. when using a transaction, choose a broker and use it for the entire transaction (unless the broker dies). When sending a MessageAck use the broker that sent the original message etc.
        Hide
        Daniel Rodrigues added a comment -

        Hi,

        would this include also tackling the problem that having a persistent consumer on one broker he won't be forwarded the messages that were persisted in some other?

        I am thinking about the following case: we have a network of brokers, say brokerA and brokerB. A consumer subscribes with a unique id client1:subscription1 to brokerB. Therefore, if offline, messages will be persisted on brokerB. After a while, brokerB goes down, and the same consumer fails over to brokerA. He's able to receive all new messages, but will not receive persisted messages when brokerB comes back to life.

        What I would like to see is basically a consumer be recognized as unique over an entire network of brokers and not just on one broker, so messages persisted in one broker could also be forwarded whenever a durable subscription would appear somewhere else on the network.

        Thank you!
        daniel

        Show
        Daniel Rodrigues added a comment - Hi, would this include also tackling the problem that having a persistent consumer on one broker he won't be forwarded the messages that were persisted in some other? I am thinking about the following case: we have a network of brokers, say brokerA and brokerB. A consumer subscribes with a unique id client1:subscription1 to brokerB. Therefore, if offline, messages will be persisted on brokerB. After a while, brokerB goes down, and the same consumer fails over to brokerA. He's able to receive all new messages, but will not receive persisted messages when brokerB comes back to life. What I would like to see is basically a consumer be recognized as unique over an entire network of brokers and not just on one broker, so messages persisted in one broker could also be forwarded whenever a durable subscription would appear somewhere else on the network. Thank you! daniel
        Hide
        james strachan added a comment -

        Daniel: the jedi transport is different to the store and forward networks. Store and forward networks are designed to forward subscription information across a network of brokers - to do exactly what you describe. So just using a store and foward network should do the trick for your requirement.

        The jedi transport is an alternative to broker store-forward networks; where brokers don't know anything about each other - and the clients do all the work. e.g. producers send to any available broker and the consumers consume from all the brokers silently under the covers. Over time this could get clever, load balancing traffic over destinations evenly - or partitioning destinations to different brokers etc

        Show
        james strachan added a comment - Daniel: the jedi transport is different to the store and forward networks. Store and forward networks are designed to forward subscription information across a network of brokers - to do exactly what you describe. So just using a store and foward network should do the trick for your requirement. The jedi transport is an alternative to broker store-forward networks; where brokers don't know anything about each other - and the clients do all the work. e.g. producers send to any available broker and the consumers consume from all the brokers silently under the covers. Over time this could get clever, load balancing traffic over destinations evenly - or partitioning destinations to different brokers etc

          People

          • Assignee:
            Rob Davies
            Reporter:
            james strachan
          • Votes:
            4 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development