Uploaded image for project: 'Derby'
  1. Derby
  2. DERBY-6879

Engine deadlock between XA timeout handling and cleanupOnError

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 10.10.2.0
    • 10.13.1.0
    • Services
    • None
    • Solaris 10.5 on Oracle M5000
    • Crash, Data corruption
    • Patch

    Description

      Deadlock between XA timer cleanup task and the ContextManager.cleanupOnError

      Found one Java-level deadlock:
      =============================
      "DRDAConnThread_34":
      waiting to lock monitor 0x0000000104b14d18 (object 0xfffffffd9090f058, a org.apache.derby.jdbc.XATransactionState),
      which is held by "Timer-0"
      "Timer-0":
      waiting to lock monitor 0x00000001038b96e8 (object 0xfffffffd9090d8b0, a org.apache.derby.impl.jdbc.EmbedConnection40),
      which is held by "DRDAConnThread_34"

      Java stack information for the threads listed above:
      ===================================================
      "DRDAConnThread_34":
      at org.apache.derby.jdbc.XATransactionState.cleanupOnError(Unknown Source)

      • waiting to lock <0xfffffffd9090f058> (a org.apache.derby.jdbc.XATransactionState)
        at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
      • locked <0xfffffffd9090d8b0> (a org.apache.derby.impl.jdbc.EmbedConnection40)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(Unknown Source)
        at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSTTobjects(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSTT(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
        "Timer-0":
        at org.apache.derby.impl.jdbc.EmbedConnection.xa_rollback(Unknown Source)
      • waiting to lock <0xfffffffd9090d8b0> (a org.apache.derby.impl.jdbc.EmbedConnection40)
        at org.apache.derby.jdbc.XATransactionState.cancel(Unknown Source)
      • locked <0xfffffffd9090f058> (a org.apache.derby.jdbc.XATransactionState)
        at org.apache.derby.jdbc.XATransactionState$CancelXATransactionTask.run(Unknown Source)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

      Found 1 deadlock.

      This deadlock caused Derby to create 18000 transaction recovery logs because of the XA transaction that did not cleanup in the timeout. Rebooting the system would cause a 50 hour boot up time to process the transaction logs so recovery had to be done by going to a backup database before the issue occurred.

      Attachments

        1. derby-6879-2016-07-05.diff
          29 kB
          Brett Bergquist
        2. derby-6879-2016-07-08.diff
          29 kB
          Brett Bergquist
        3. derby-6879-2016-07-17.diff
          17 kB
          Brett Bergquist
        4. derby-6879-test.diff
          24 kB
          Brett Bergquist
        5. diff-0717-ignore-whitespace.diff
          6 kB
          Bryan Pendleton
        6. svnstatus.txt
          0.4 kB
          Brett Bergquist
        7. testFail.zip
          455 kB
          Bryan Pendleton

        Activity

          People

            bbergquist Brett Bergquist
            bbergquist Brett Bergquist
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: