Qpid
  1. Qpid
  2. QPID-5397

Dispatch - Crash occurs when linked deliveries are concurrently settled

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Qpid Dispatch
    • Labels:
      None

      Description

      Dispatch router crashed while running a test in which a client/server pair exchanges requests and responses with acknowledgement (not pre-settled). The test does not throttle the message rates.

      After several tens-of-thousands of transfers, the router crashed while settling a delivery and attempting to unlink it from its peer. Apparently two linked deliveries were concurrently settled on separate threads.

        Activity

        Hide
        ASF subversion and git services added a comment -

        Commit 1548737 from Ted Ross in branch 'dispatch/trunk'
        [ https://svn.apache.org/r1548737 ]

        QPID-5397

        • Redesigned the fix to the mutex problem
        • Added a test to detect the settlement-collision problem
        • Increased the verbosity of the system tests in qdtest
        • Some minor cleanup in the python router and qdstat
        Show
        ASF subversion and git services added a comment - Commit 1548737 from Ted Ross in branch 'dispatch/trunk' [ https://svn.apache.org/r1548737 ] QPID-5397 Redesigned the fix to the mutex problem Added a test to detect the settlement-collision problem Increased the verbosity of the system tests in qdtest Some minor cleanup in the python router and qdstat
        Hide
        Ted Ross added a comment -

        The fix was an improvement, but there is still a window of opportunity for a problem.

        If a delivery is sent unsettled, and is then settled by the sender before being settled by the receiver (due to exceeding the outgoing window in Messenger for example), Dispatch may attempt to re-settle the sender-side delivery when the receiver settles thus causing problems in Proton Engine and possibly double-freeing the structure.

        A new fix will hold off the unlinking and freeing of linked deliveries until they are both settled.

        Show
        Ted Ross added a comment - The fix was an improvement, but there is still a window of opportunity for a problem. If a delivery is sent unsettled, and is then settled by the sender before being settled by the receiver (due to exceeding the outgoing window in Messenger for example), Dispatch may attempt to re-settle the sender-side delivery when the receiver settles thus causing problems in Proton Engine and possibly double-freeing the structure. A new fix will hold off the unlinking and freeing of linked deliveries until they are both settled.
        Hide
        ASF subversion and git services added a comment -

        Commit 1548209 from Ted Ross in branch 'dispatch/trunk'
        [ https://svn.apache.org/r1548209 ]

        QPID-5397 - Added a global lock to protect the linking and unlinking of peered deliveries.

        Show
        ASF subversion and git services added a comment - Commit 1548209 from Ted Ross in branch 'dispatch/trunk' [ https://svn.apache.org/r1548209 ] QPID-5397 - Added a global lock to protect the linking and unlinking of peered deliveries.

          People

          • Assignee:
            Ted Ross
            Reporter:
            Ted Ross
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development