Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-5139

HA transactions block a thread, can deadlock the broker.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.24
    • Fix Version/s: 0.25
    • Component/s: C++ Clustering
    • Labels:
      None

      Description

      When the client sends a "prepare" command for a transaction, the thread handling that command is blocked until all backups have responded with their prepare status. This can easily deadlock the broker if there are more concurrent transactions than worker threads.

      To reproduce you need a way to delay completion of a transaction. The following regression test illustrates, it will be added to TransactionTests in ha_tests.py as part of the fix:

          def test_tx_block_threads(self):
              """Verify that TXs blocked in commit don't block broker threads."""
              cluster = HaCluster(self, 2, args=["--worker-threads=2"])
              sessions = [cluster[0].connect().session(transactional=True) for i in xrange(2)]
              for s in sessions: s.sender("foo;{create:always}").send("foo")
              self.assertEqual(2, len(cluster[1].agent().tx_queues()))
              os.kill(cluster[1].pid, signal.SIGSTOP) # Freeze backup so tx can't complete.
              threads = [ Thread(target=s.commit) for s in sessions]
              for t in threads: t.start()
              cluster[0].ready(timeout=1) # Should not block
              os.kill(cluster[1].pid, signal.SIGCONT) # Allow tx to complete.
              for t in threads: t.join()
              c.close()
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                aconway Alan Conway
                Reporter:
                aconway Alan Conway
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: