Uploaded image for project: 'Derby'
  1. Derby
  2. DERBY-5552

Derby threads hanging when using ClientXADataSource and a deadlock or lock timeout occurs

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Urgent
    • Patch Available, Repro attached
    • Crash, Seen in production

    Description

      The issue arrives when multiple XA transactions are done in parallel and there is either a lock timeout or a lock deadlock detected. When this happens the connection is leaked in the Glassfish connection pool and the client thread hangs in "org.apache.derby.client.netReply.fill(Reply.java:172)".

      Shutting down the app server fails because the thread has a lock in "org.apache.derby.client.net.NetConnection40" and another task is calling "org.apache.derby.client.ClientPooledConnection.close(ClientPooledConnection.java:214)" which is waiting for the lock.

      Killing the appsever using "kill" and then attempting to shutdown Derby network server causes the Network Server to hang. One of the threads hangs waiting for a lock at "org.apache.derby.impl.drda.NeworkServerControlImpl.removeFromSessionTable(NetworkServerControlImpl.java:1525)" and the "main" thread has this locked at "org.apache.derby.impl.drda.NetworkServerControlImpl.executeWork(NetworkServerControlImpl.java:2242)" and it itself is waiting for a lock which belongs to a thread that is stuck at "org.apache.derby.impl.services.locks.ActiveLock.waitForGrant(ActiveLock.java:118) which is in the TIMED_WAITING state.

      Only by killing the Network Server using "kill" is possible at this point.

      There are transactions left even though all clients have been removed.

      Attachments

        1. appserverstack.txt
          63 kB
          Brett Bergquist
        2. client.tar.Z
          69 kB
          Brett Bergquist
        3. derby.log
          320 kB
          Brett Bergquist
        4. derby-5552_withexpanded_test_diff.txt
          9 kB
          Katherine Marsden
        5. derby-5552_withtest_diff.txt
          5 kB
          Katherine Marsden
        6. derby-5552_withtest_diff.txt
          5 kB
          Katherine Marsden
        7. DERBY-5552-p1.patch
          2 kB
          Brett Bergquist
        8. DERBY-5552-p2.patch
          0.5 kB
          Brett Bergquist
        9. derbystackatshutdown.txt
          29 kB
          Brett Bergquist
        10. execute.patch
          1 kB
          Brett Bergquist
        11. ReproDerby5552DB2.java
          4 kB
          Katherine Marsden
        12. ReproDerby5552LockTimeout.java
          5 kB
          Katherine Marsden
        13. transactionsleft.txt
          4 kB
          Brett Bergquist
        14. utilXid.java
          2 kB
          Katherine Marsden

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            brettw Brett Wooldridge
            bbergquist Brett Bergquist
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment