[QPID-4036] Failed client connections permanently exhaust broker's max connections limit

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.16
• Fix Version/s: 0.18
• Component/s: C++ Broker
• Labels: sasl ssl

Environment

CentOS release 5.5 (Final)
Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48)

• Assignee: Unassigned
• Reporter: Paul Colby
• Created: 01/Jun/12 01:45
• Updated: 29/Jul/13 18:53
• Resolved: 18/Jul/12 16:59

Description

I'm running a set of Qpid 0.16 C++ brokers with configuration like:

cluster-name="mm-queue-cluster"
cluster-cman=yes
cluster-mechanism=PLAIN
cluster-username=broker
cluster-password=abc123
cluster-url=ssl:gateway02:5671

auth=yes

ssl-cert-db=/etc/qpid/certs/broker
ssl-cert-password-file=/etc/qpid/certs/pass.txt
ssl-cert-name=broker.messagemedia.com.au
require-encryption=yes

ie the broker is requiring both encryption and authentication (configured SASL mech list is CRAM-MD5 DIGEST-MD5 EXTERNAL PLAIN).

Now, if a client (let's use qpid-stat for example) connects via SSL (amqps) and authenticates successfully, then everything is happy.

However, if a client repeatedly fails to use SSL and/or fails to provide credentials, then the broker loses one of it's configured max connections every time!

So, for example, if we start the broker using the configuration shown above, then do this:

for i in `seq 1 550`; do echo $i; qpid-stat -q ; done

The above loop will report ~ 500 AuthenticationFailure errors, then switch to ConnectionError errors. Once the ConnectionError errors begin, all further connections to the broker will be rejected - permanently (until the broker is restarted), with the broker logging:

error Client max connection count limit exceeded: 500 connection refused

From my testing, the following loops never cause an issue (with this configuration):

for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/guest@localhost -q ; done # Works as expected.
for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://guest/wrong@localhost -q ; done # AuthenticationFailure as expected.

Whereas any of the following will break the broker:

for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/guest@localhost -q ; done # AuthenticationFailure, then ConnectionError.
for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://guest/wrong@localhost -q ; done # AuthenticationFailure, then ConnectionError.
for i in `seq 1 550`; do echo $i; qpid-stat -b amqp://localhost -q ; done             # AuthenticationFailure, then ConnectionError.
for i in `seq 1 550`; do echo $i; qpid-stat -b amqps://localhost -q ; done            # AuthenticationFailure, then ConnectionError.

Issue Links

• is duplicated by: QPID-4129 C++ Broker failed client authentication in multi-node cluster causes connection leak