Uploaded image for project: 'Derby'
  1. Derby
  2. DERBY-6879

Engine deadlock between XA timeout handling and cleanupOnError

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.10.2.0
    • Fix Version/s: 10.13.1.0
    • Component/s: Services
    • Labels:
      None
    • Environment:
      Solaris 10.5 on Oracle M5000
    • Bug behavior facts:
      Crash, Data corruption
    • Flags:
      Patch

      Description

      Deadlock between XA timer cleanup task and the ContextManager.cleanupOnError

      Found one Java-level deadlock:
      =============================
      "DRDAConnThread_34":
      waiting to lock monitor 0x0000000104b14d18 (object 0xfffffffd9090f058, a org.apache.derby.jdbc.XATransactionState),
      which is held by "Timer-0"
      "Timer-0":
      waiting to lock monitor 0x00000001038b96e8 (object 0xfffffffd9090d8b0, a org.apache.derby.impl.jdbc.EmbedConnection40),
      which is held by "DRDAConnThread_34"

      Java stack information for the threads listed above:
      ===================================================
      "DRDAConnThread_34":
      at org.apache.derby.jdbc.XATransactionState.cleanupOnError(Unknown Source)

      • waiting to lock <0xfffffffd9090f058> (a org.apache.derby.jdbc.XATransactionState)
        at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
      • locked <0xfffffffd9090d8b0> (a org.apache.derby.impl.jdbc.EmbedConnection40)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedPreparedStatement.execute(Unknown Source)
        at org.apache.derby.iapi.jdbc.BrokeredPreparedStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAStatement.execute(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSTTobjects(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.parseEXCSQLSTT(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.processCommands(Unknown Source)
        at org.apache.derby.impl.drda.DRDAConnThread.run(Unknown Source)
        "Timer-0":
        at org.apache.derby.impl.jdbc.EmbedConnection.xa_rollback(Unknown Source)
      • waiting to lock <0xfffffffd9090d8b0> (a org.apache.derby.impl.jdbc.EmbedConnection40)
        at org.apache.derby.jdbc.XATransactionState.cancel(Unknown Source)
      • locked <0xfffffffd9090f058> (a org.apache.derby.jdbc.XATransactionState)
        at org.apache.derby.jdbc.XATransactionState$CancelXATransactionTask.run(Unknown Source)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

      Found 1 deadlock.

      This deadlock caused Derby to create 18000 transaction recovery logs because of the XA transaction that did not cleanup in the timeout. Rebooting the system would cause a 50 hour boot up time to process the transaction logs so recovery had to be done by going to a backup database before the issue occurred.

        Attachments

        1. derby-6879-2016-07-05.diff
          29 kB
          Brett Bergquist
        2. derby-6879-2016-07-08.diff
          29 kB
          Brett Bergquist
        3. derby-6879-2016-07-17.diff
          17 kB
          Brett Bergquist
        4. derby-6879-test.diff
          24 kB
          Brett Bergquist
        5. diff-0717-ignore-whitespace.diff
          6 kB
          Bryan Pendleton
        6. svnstatus.txt
          0.4 kB
          Brett Bergquist
        7. testFail.zip
          455 kB
          Bryan Pendleton

          Activity

            People

            • Assignee:
              bbergquist Brett Bergquist
              Reporter:
              bbergquist Brett Bergquist
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: