Qpid
  1. Qpid
  2. QPID-4036

Failed client connections permanently exhaust broker's max connections limit

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.16
    • Fix Version/s: 0.18
    • Component/s: C++ Broker
    • Labels:
    • Environment:

      CentOS release 5.5 (Final)
      Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
      gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48)

      Description

      I'm running a set of Qpid 0.16 C++ brokers with configuration like:

      cluster-name="mm-queue-cluster"
      cluster-cman=yes
      cluster-mechanism=PLAIN
      cluster-username=broker
      cluster-password=abc123
      cluster-url=ssl:gateway02:5671
      
      auth=yes
      
      ssl-cert-db=/etc/qpid/certs/broker
      ssl-cert-password-file=/etc/qpid/certs/pass.txt
      ssl-cert-name=broker.messagemedia.com.au
      require-encryption=yes
      

      ie the broker is requiring both encryption and authentication (configured SASL mech list is CRAM-MD5 DIGEST-MD5 EXTERNAL PLAIN).

      Now, if a client (let's use qpid-stat for example) connects via SSL (amqps) and authenticates successfully, then everything is happy.

      However, if a client repeatedly fails to use SSL and/or fails to provide credentials, then the broker loses one of it's configured max connections every time!

      So, for example, if we start the broker using the configuration shown above, then do this:

      for i in `seq 1 550`; do echo $i; qpid-stat -q ; done

      The above loop will report ~ 500 AuthenticationFailure errors, then switch to ConnectionError errors. Once the ConnectionError errors begin, all further connections to the broker will be rejected - permanently (until the broker is restarted), with the broker logging:

      error Client max connection count limit exceeded: 500 connection refused

      From my testing, the following loops never cause an issue (with this configuration):

      for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/guest@localhost -q ; done # Works as expected.
      for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/wrong@localhost -q ; done # AuthenticationFailure as expected.
      

      Whereas any of the following will break the broker:

      for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/guest@localhost -q ; done # AuthenticationFailure, then ConnectionError.
      for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/wrong@localhost -q ; done # AuthenticationFailure, then ConnectionError.
      for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://localhost -q ; done             # AuthenticationFailure, then ConnectionError.
      for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://localhost -q ; done            # AuthenticationFailure, then ConnectionError.
      

        Issue Links

          Activity

          Paul Colby created issue -
          Paul Colby made changes -
          Field Original Value New Value
          Summary Failed client connections permanently exhausting broker's max connections limit Failed client connections permanently exhaust broker's max connections limit
          Hide
          Andy Goldstein added a comment -

          This sounds similar to QPID-3438, which was supposed to be fixed in 0.16...

          Show
          Andy Goldstein added a comment - This sounds similar to QPID-3438 , which was supposed to be fixed in 0.16...
          Hide
          Paul Colby added a comment -

          Thanks Andy.

          Yes QPID-3438 definitely looks like the same issue. However, as you said, that was supposedly fixed in 0.16, which I'm using.

          A quick survey shows that most (all?) of the commits for QPID-3438 (ie r1215127) are present in 0.16 as expected, however some of those changes have since been changed further (eg cluster/Connection.cppp Connection::connectionError).

          So, perhaps this issues is a regression against the fixes that were done for QPID-3438 (doesn't look likely), or another corner case that wasn't covered by QPID-3438 (more likely?).

          Show
          Paul Colby added a comment - Thanks Andy. Yes QPID-3438 definitely looks like the same issue. However, as you said, that was supposedly fixed in 0.16, which I'm using. A quick survey shows that most (all?) of the commits for QPID-3438 (ie r1215127) are present in 0.16 as expected, however some of those changes have since been changed further (eg cluster/Connection.cppp Connection::connectionError). So, perhaps this issues is a regression against the fixes that were done for QPID-3438 (doesn't look likely), or another corner case that wasn't covered by QPID-3438 (more likely?).
          Hide
          Paul Colby added a comment -

          I've done a quick review of the changes from r1215127 that have since been dropped, and it seems unlikely that the subsequent changes would have regresses QPID-3438.

          Also, I've found a simpler test case:

          1. Compile and install 0.16 from source; make sure CPG is enabled (ie install openais or corosync).
          2. Configure SASL to allow only PLAIN (or at least disable ANONYMOUS).
          3. Run: qpidd --cluster-name=test
            • you'll probably need to run it as root, or some other user with permission to access the underlying cluster services.
          4. Run: for i in `seq 1 550`; do echo $i; qpid-stat -q; done

          That's it... sit back and watch the broker become unusable

          Incidentally, the problem does not occur if the broker is not instructed to use clustering.

          Show
          Paul Colby added a comment - I've done a quick review of the changes from r1215127 that have since been dropped, and it seems unlikely that the subsequent changes would have regresses QPID-3438 . Also, I've found a simpler test case: Compile and install 0.16 from source; make sure CPG is enabled (ie install openais or corosync). Configure SASL to allow only PLAIN (or at least disable ANONYMOUS). Run: qpidd --cluster-name=test you'll probably need to run it as root , or some other user with permission to access the underlying cluster services. Run: for i in `seq 1 550`; do echo $i; qpid-stat -q; done That's it... sit back and watch the broker become unusable Incidentally, the problem does not occur if the broker is not instructed to use clustering.
          Gordon Sim made changes -
          Link This issue is duplicated by QPID-4129 [ QPID-4129 ]
          Hide
          Justin Ross added a comment -

          Reviewed by Alan. Approved for 0.18. Corresponding commit is http://svn.apache.org/viewvc?view=revision&revision=1360214 .

          Show
          Justin Ross added a comment - Reviewed by Alan. Approved for 0.18. Corresponding commit is http://svn.apache.org/viewvc?view=revision&revision=1360214 .
          Hide
          Chuck Rolke added a comment -

          r1360214

          Show
          Chuck Rolke added a comment - r1360214
          Chuck Rolke made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.18 [ 12322451 ]
          Resolution Fixed [ 1 ]
          Justin Ross made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Paul Colby
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development