Qpid
  1. Qpid
  2. QPID-4144

HA broker deadlocks on broeker::QueueRegistry lock and ha::Primary lock

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.17
    • Fix Version/s: 0.18
    • Component/s: C++ Clustering
    • Labels:
      None

      Description

      Running tests repeatedly, the broker deadlocked with the attached stack trace.

      The problem call sequences are:
      1. QueueRegistry::destroy takes QueuerRegistry lock > ConfigurationObserver::queueDestroy > ha::Primary::queueDestroy takes Primary lock.
      2. ConnectionObserver::opened cals Primary::opened lock> RemoteBackup>getQueues().eachQueue

      This patch breaks the deadlock at both ends: QueueRegistry no longer holds the lock across the observer call and Primary does not hold the lock across eachQueue.

      1. thread-dump.txt
        23 kB
        Alan Conway

        Activity

        Hide
        Alan Conway added a comment -

        On 0.18 branch:
        ------------------------------------------------------------------------
        r1363486 | aconway | 2012-07-19 15:16:12 -0400 (Thu, 19 Jul 2012) | 10 lines

        QPID-4144 HA broker deadlocks on broker::QueueRegistry lock and ha::Primary lock

        Running tests repeatedly, the broker deadlocked with the attached stack trace.

        The problem call sequences are:
        1. QueueRegistry::destroy takes QueuerRegistry lock > ConfigurationObserver::queueDestroy > ha::Primary::queueDestroy takes Primary lock.
        2. ConnectionObserver::opened cals Primary::opened lock> RemoteBackup>getQueues().eachQueue

        This patch breaks the deadlock at both ends: QueueRegistry no longer holds the lock across the observer call and Primary does not hold the lock across eachQueue.

        Show
        Alan Conway added a comment - On 0.18 branch: ------------------------------------------------------------------------ r1363486 | aconway | 2012-07-19 15:16:12 -0400 (Thu, 19 Jul 2012) | 10 lines QPID-4144 HA broker deadlocks on broker::QueueRegistry lock and ha::Primary lock Running tests repeatedly, the broker deadlocked with the attached stack trace. The problem call sequences are: 1. QueueRegistry::destroy takes QueuerRegistry lock > ConfigurationObserver::queueDestroy > ha::Primary::queueDestroy takes Primary lock. 2. ConnectionObserver::opened cals Primary::opened lock> RemoteBackup>getQueues().eachQueue This patch breaks the deadlock at both ends: QueueRegistry no longer holds the lock across the observer call and Primary does not hold the lock across eachQueue.
        Hide
        Justin Ross added a comment -

        Reviewed by Gordon. Approved for 0.18.

        Show
        Justin Ross added a comment - Reviewed by Gordon. Approved for 0.18.
        Hide
        Gordon Sim added a comment -

        Looks good to me; I would approve of merging this onto 0.18 release branch.

        Show
        Gordon Sim added a comment - Looks good to me; I would approve of merging this onto 0.18 release branch.
        Hide
        Alan Conway added a comment -

        Fixed on trunk, request merge to 0.18 branch.

        ------------------------------------------------------------------------
        r1362584 | aconway | 2012-07-17 13:39:54 -0400 (Tue, 17 Jul 2012) | 10 lines

        QPID-4144 HA broker deadlocks on broker::QueueRegistry lock and ha::Primary lock

        Running tests repeatedly, the broker deadlocked with the attached stack trace.

        The problem call sequences are:
        1. QueueRegistry::destroy takes QueuerRegistry lock > ConfigurationObserver::queueDestroy > ha::Primary::queueDestroy takes Primary lock.
        2. ConnectionObserver::opened cals Primary::opened lock> RemoteBackup>getQueues().eachQueue

        This patch breaks the deadlock at both ends: QueueRegistry no longer holds the lock across the observer call and Primary does not hold the lock across eachQueue.

        ------------------------------------------------------------------------

        Show
        Alan Conway added a comment - Fixed on trunk, request merge to 0.18 branch. ------------------------------------------------------------------------ r1362584 | aconway | 2012-07-17 13:39:54 -0400 (Tue, 17 Jul 2012) | 10 lines QPID-4144 HA broker deadlocks on broker::QueueRegistry lock and ha::Primary lock Running tests repeatedly, the broker deadlocked with the attached stack trace. The problem call sequences are: 1. QueueRegistry::destroy takes QueuerRegistry lock > ConfigurationObserver::queueDestroy > ha::Primary::queueDestroy takes Primary lock. 2. ConnectionObserver::opened cals Primary::opened lock> RemoteBackup>getQueues().eachQueue This patch breaks the deadlock at both ends: QueueRegistry no longer holds the lock across the observer call and Primary does not hold the lock across eachQueue. ------------------------------------------------------------------------
        Hide
        Alan Conway added a comment -

        Thread dump of the deadlocked process.

        Show
        Alan Conway added a comment - Thread dump of the deadlocked process.

          People

          • Assignee:
            Alan Conway
            Reporter:
            Alan Conway
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development