ActiveMQ
  1. ActiveMQ
  2. AMQ-3161

Race condition in ActiveMQ Journal Checkpoint worker thread cleanup leads to multiple running instances

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 5.4.2
    • Fix Version/s: 5.5.0
    • Component/s: Message Store
    • Labels:
      None
    • Environment:

      CentOS 5.2

    • Patch Info:
      Patch Available

      Description

      If the ActiveMQ Journal Checkpoint worker thread dies for any reason, any updates to KahaDB trigger the thread to be restarted. Unfortunately, there is a race condition between the Thread.isAlive() check and the restart of the thread, potentially leading to multiple instances of the thread being spawned. We have observed in a busy environment as many as 8-10 instances of this thread, all waking up every 5 seconds to perform database checkpoints, bringing throughput on the broker to a halt.

      The attached patch adds a lock around access to creation/destruction of this thread and prevents a second instance from being launched inadvertently.

        Issue Links

          Activity

          Hide
          Craig Condit added a comment -

          This patch to MessageDatabase.java prevents accidental launching of multiple Journal checkpoint worker threads.

          Show
          Craig Condit added a comment - This patch to MessageDatabase.java prevents accidental launching of multiple Journal checkpoint worker threads.
          Hide
          Craig Condit added a comment -

          AMQ-3160 is the root cause of the thread aborting, leading to the case where this fix is necessary.

          Show
          Craig Condit added a comment - AMQ-3160 is the root cause of the thread aborting, leading to the case where this fix is necessary.
          Hide
          Gary Tully added a comment -

          patch applied with thanks in r1063710

          URL: http://svn.apache.org/viewvc?rev=1063710&view=rev

          one minor mod, left unsynced check in store, such that locking overhead occurs only if thread needs a restart.

          Show
          Gary Tully added a comment - patch applied with thanks in r1063710 URL: http://svn.apache.org/viewvc?rev=1063710&view=rev one minor mod, left unsynced check in store, such that locking overhead occurs only if thread needs a restart.

            People

            • Assignee:
              Gary Tully
              Reporter:
              Craig Condit
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development