Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-5125

Broker and clients hang

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 5.9.0, 5.10.0
    • Fix Version/s: 5.10.1, 5.11.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows 7, Linux
      JAVA_HOME=C:\Program Files\Java\jdk1.7.0_07
      ActiveMQ 5.9.0 with LevelDB storage adapter enabled

      Description

      JMS clients start to hang after a while in calls such as session.createObjectMessage(). Both the broker and the hanging clients can't be easily shut down when this happens - only forcefully applied kill's do the job.

      I'm using queues and transactional sessions. All clients (producers and consumers) are in the same Java VM. There is only one JMS connection between the application and the broker. Each client has its own session, but they all share the same connection.

      Normally, the data directory of the LevelDb contains only a few log files. But in my case, the number of log files is steadily increasing.

      Furthermore, I was able to track down the issue to following circumstance: The problem only occurs, when consumers do a rollback instead of a commit when they receive the message. The rollback / redelivery works as expected - the same message is received again after a previous rollback.

      As far as I can tell, the problem does not occur with KahaDb.

      I'll attach a test program that provokes the error. It sets up a few hundred queues, consumers and producers. The consumers just receive the message and commit the session, but they also do "random" rollbacks. It can be observed immediately that the number of files starts increasing in the data directory. After a few minutes, the clients hang - sometimes sooner, sometimes later. I'll also attach the config file for the broker.

      I am aware, that heavy rollbacking should not happen in normal operation. But from a long term stability perspective, this is a blocker for us.

        Attachments

        1. activemq.xml
          6 kB
          Albert Barmettler
        2. AMQ-5125_made_synchronization_in_LevelDBStore_use_private_lock.5.9.2.patch
          3 kB
          James Home
        3. Broker-threaddump-1396544622970.tdump
          47 kB
          Albert Barmettler
        4. broker-threads-blocked.tdump
          77 kB
          Tomas Klubal
        5. Clients-threaddump-1396544622970.tdump
          299 kB
          Albert Barmettler
        6. src.zip
          6 kB
          Albert Barmettler
        7. VM.PNG
          46 kB
          Albert Barmettler

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Albert Barmettler Albert Barmettler
            • Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: