ActiveMQ
  1. ActiveMQ
  2. AMQ-3656

Activemq hung or slows down significantly when high number of connections occurs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: 5.5.0
    • Fix Version/s: None
    • Component/s: Performance Test
    • Labels:
      None
    • Environment:

      Linux AS5 (CentOS release 5.5),
      Java: SUNJAVA2 1.6.0r14a

      Description

      We have an issue with AMQ performance. In our test program, we run two amq nodes in a network cluster. Both AMQ client and server/worker consist of producers and consumers. When client makes a connection to an AMQ node, it will send messages to three queues and meanwhile listen to 2 topics and one temporary queue. In AMQ worker/server side, it will listen to three queues and send messages back to corresponding two topics and one temp queue. During our test, We first had 10 servers/workers running at the same node as one AMQ broker was running, then started client program remotely by making 50 connections first, then repeated the 50 connections again and again. We found when the connection number reached above 650, AMQ slowed down significantly, as when we tried to send a text message through Jconsole (which run remotely also); we couldn't get a response from Jconsole with a returned messageID. Any message sent during that time was not enqueued in time. Looks like there was a blocking to the producer side, but we had producer flow control turned off. The heap memory usage looked fine to us during that time (well below 3G we assigned for JVM). But we did notice that we even couldn't do a thread dump during that time, as it kept failing to attach the process. It took some time (from 10 seconds to 15 minutes or more) for the AMQ to come back and response to the messages sent, and by then we could do thread dump without a problem.
      In amq configure side, I have followed suggestion by Dave Stanley http://www.pepperdust.org/?p=150. Not sure why we still have such a "hung" problem? I'll attach the test case program with our amq configure file latter.

        Activity

        Hide
        Yelu huang added a comment -

        activemq.xml files are include in the attached pkg. check activemq script (also attached) for JVM options. run runpltsvr.ksh in the same node AMQ broker is running, then run runpltclt.ksh twice remotely. Run Jconsole remotely and try to send a text message to any queue, see the response from Jconsole when the high number of connection is reached.

        Show
        Yelu huang added a comment - activemq.xml files are include in the attached pkg. check activemq script (also attached) for JVM options. run runpltsvr.ksh in the same node AMQ broker is running, then run runpltclt.ksh twice remotely. Run Jconsole remotely and try to send a text message to any queue, see the response from Jconsole when the high number of connection is reached.
        Hide
        Timothy Bish added a comment -

        You should try testing against a 5.6-SNAPSHOT for comparison as there have been a number of fixes since 5.5.0

        Show
        Timothy Bish added a comment - You should try testing against a 5.6-SNAPSHOT for comparison as there have been a number of fixes since 5.5.0
        Hide
        Timothy Bish added a comment -

        Not logs for configurations provided, not tests cases or other materials to properly analyze this to determine if there is a real problem or if the broker is just utilizing all available resources per configuration.

        Show
        Timothy Bish added a comment - Not logs for configurations provided, not tests cases or other materials to properly analyze this to determine if there is a real problem or if the broker is just utilizing all available resources per configuration.

          People

          • Assignee:
            Unassigned
            Reporter:
            Yelu huang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development