Derby
  1. Derby
  2. DERBY-5552

Derby threads hanging when using ClientXADataSource and a deadlock or lock timeout occurs

    Details

    • Urgency:
      Urgent
    • Issue & fix info:
      Patch Available, Repro attached
    • Bug behavior facts:
      Crash, Seen in production

      Description

      The issue arrives when multiple XA transactions are done in parallel and there is either a lock timeout or a lock deadlock detected. When this happens the connection is leaked in the Glassfish connection pool and the client thread hangs in "org.apache.derby.client.netReply.fill(Reply.java:172)".

      Shutting down the app server fails because the thread has a lock in "org.apache.derby.client.net.NetConnection40" and another task is calling "org.apache.derby.client.ClientPooledConnection.close(ClientPooledConnection.java:214)" which is waiting for the lock.

      Killing the appsever using "kill" and then attempting to shutdown Derby network server causes the Network Server to hang. One of the threads hangs waiting for a lock at "org.apache.derby.impl.drda.NeworkServerControlImpl.removeFromSessionTable(NetworkServerControlImpl.java:1525)" and the "main" thread has this locked at "org.apache.derby.impl.drda.NetworkServerControlImpl.executeWork(NetworkServerControlImpl.java:2242)" and it itself is waiting for a lock which belongs to a thread that is stuck at "org.apache.derby.impl.services.locks.ActiveLock.waitForGrant(ActiveLock.java:118) which is in the TIMED_WAITING state.

      Only by killing the Network Server using "kill" is possible at this point.

      There are transactions left even though all clients have been removed.

      1. appserverstack.txt
        63 kB
        Brett Bergquist
      2. client.tar.Z
        69 kB
        Brett Bergquist
      3. derby.log
        320 kB
        Brett Bergquist
      4. derby-5552_withexpanded_test_diff.txt
        9 kB
        Kathey Marsden
      5. derby-5552_withtest_diff.txt
        5 kB
        Kathey Marsden
      6. derby-5552_withtest_diff.txt
        5 kB
        Kathey Marsden
      7. DERBY-5552-p1.patch
        2 kB
        Brett Bergquist
      8. DERBY-5552-p2.patch
        0.5 kB
        Brett Bergquist
      9. derbystackatshutdown.txt
        29 kB
        Brett Bergquist
      10. execute.patch
        1 kB
        Brett Bergquist
      11. ReproDerby5552DB2.java
        4 kB
        Kathey Marsden
      12. ReproDerby5552LockTimeout.java
        5 kB
        Kathey Marsden
      13. transactionsleft.txt
        4 kB
        Brett Bergquist
      14. utilXid.java
        2 kB
        Kathey Marsden

        Issue Links

          Activity

          Brett Bergquist created issue -
          Brett Bergquist made changes -
          Field Original Value New Value
          Attachment derby.log [ 12508400 ]
          Brett Bergquist made changes -
          Attachment client.tar.Z [ 12508401 ]
          Brett Bergquist made changes -
          Attachment transactionsleft.txt [ 12508402 ]
          Brett Bergquist made changes -
          Attachment appserverstack.txt [ 12508403 ]
          Brett Bergquist made changes -
          Attachment derbystackatshutdown.txt [ 12508404 ]
          Brett Bergquist made changes -
          Attachment execute.patch [ 12508456 ]
          Kathey Marsden made changes -
          Link This issue relates to DERBY-5553 [ DERBY-5553 ]
          Brett Bergquist made changes -
          Attachment DERBY-5552-p1.patch [ 12509290 ]
          Brett Bergquist made changes -
          Attachment DERBY-5552-p2.patch [ 12509291 ]
          Kathey Marsden made changes -
          Attachment ReproDerby5552LockTimeout.java [ 12509477 ]
          Dag H. Wanvik made changes -
          Issue & fix info Repro attached [ 10424 ]
          Kathey Marsden made changes -
          Assignee Kathey Marsden [ kmarsden ]
          Kathey Marsden made changes -
          Attachment derby-5552_withtest_diff.txt [ 12512868 ]
          Kathey Marsden made changes -
          Issue & fix info Repro attached [ 10424 ] Patch Available,Repro attached [ 10102,10424 ]
          Kathey Marsden made changes -
          Attachment derby-5552_withtest_diff.txt [ 12513867 ]
          Kathey Marsden made changes -
          Labels derby_triage10_9
          Urgency Urgent [ 10051 ]
          Kathey Marsden made changes -
          Attachment derby-5552_withexpanded_test_diff.txt [ 12515815 ]
          Kathey Marsden made changes -
          Attachment utilXid.java [ 12516004 ]
          Attachment ReproDerby5552DB2.java [ 12516005 ]
          Kathey Marsden made changes -
          Fix Version/s 10.9.0.0 [ 12316344 ]
          Kathey Marsden made changes -
          Affects Version/s 10.8.2.2 [ 12317968 ]
          Affects Version/s 10.7.1.1 [ 12315564 ]
          Affects Version/s 10.6.2.1 [ 12315343 ]
          Affects Version/s 10.5.3.0 [ 12314117 ]
          Affects Version/s 10.4.2.0 [ 12313345 ]
          Affects Version/s 10.3.3.0 [ 12313142 ]
          Affects Version/s 10.2.2.0 [ 12312027 ]
          Affects Version/s 10.1.3.1 [ 12311953 ]
          Kathey Marsden made changes -
          Link This issue incorporates DERBY-5633 [ DERBY-5633 ]
          Kathey Marsden made changes -
          Assignee Kathey Marsden [ kmarsden ] Brett Wooldridge [ brettw ]
          Fix Version/s 10.5.3.1 [ 12314182 ]
          Fix Version/s 10.6.2.2 [ 12315370 ]
          Fix Version/s 10.7.1.4 [ 12315902 ]
          Fix Version/s 10.8.2.3 [ 12318540 ]
          Kathey Marsden made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Kathey Marsden made changes -
          Fix Version/s 10.8.3.0 [ 12323456 ]
          Fix Version/s 10.8.2.3 [ 12318540 ]
          Gavin made changes -
          Workflow jira [ 12646885 ] Default workflow, editable Closed status [ 12802153 ]
          Knut Anders Hatlen made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Brett Wooldridge
              Reporter:
              Brett Bergquist
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development