[QPID-5294] [AMQP 1.0 JMS] Thread deadlock due to JVM bug JDK-8022788

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.24
• Fix Version/s: 0.25
• Component/s: JMS AMQP 0-x

Environment

Qpid JMS AMQP 1.0 client lib

• Assignee: Robert Godfrey
• Reporter: David Ingham
• Created: 04/Nov/13 18:56
• Updated: 19/Jul/14 11:46
• Resolved: 07/Nov/13 09:10

Description

A deadlock is occasionally observed in heavy load testing. This is apparently due to the following JVM bug:

JDK-8022788 : deadlock in SSLSocketImpl between between write and close
Affected Versions: 7u60
Fixed Versions: 7u60
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8022788

The bug manifests itself with the following deadlock:

Found one Java-level deadlock:
=============================
"Thread-392":
waiting for ownable synchronizer 0x000000078af1c4a0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "Thread-391"
"Thread-391":
waiting to lock monitor 0x00007f48d4483790 (object 0x000000078af1be60, a sun.security.ssl.SSLSocketImpl),
which is held by "Thread-392"

Java stack information for the threads listed above:
===================================================
"Thread-392":
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x000000078af1c4a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
  at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:799)
  at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:672)
  at sun.security.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:2007)
  at sun.security.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1834)
  at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1602)
• locked <0x000000078af1be60> (a sun.security.ssl.SSLSocketImpl)
  at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1538)
  at org.apache.qpid.amqp_1_0.client.Connection$2.run(Connection.java:267)
  at java.lang.Thread.run(Thread.java:724)
  "Thread-391":
  at sun.security.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:649)
• waiting to lock <0x000000078af1be60> (a sun.security.ssl.SSLSocketImpl)
  at sun.security.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1446)
  at java.net.Socket.getTcpNoDelay(Socket.java:961)
  at sun.security.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:345)
  at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:819)
  at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
  at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
• locked <0x000000078af1c570> (a sun.security.ssl.AppOutputStream)
  at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$BytesOutputHandler.processBytes(ConnectionHandler.java:420)
  at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$FrameToBytesSourceAdapter.getBytes(ConnectionHandler.java:305)
  at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$SequentialBytesSource.getBytes(ConnectionHandler.java:371)
• locked <0x000000078af1c5c8> (a org.apache.qpid.amqp_1_0.framing.ConnectionHandler$SequentialBytesSource)
  at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$BytesOutputHandler.run(ConnectionHandler.java:406)
  at java.lang.Thread.run(Thread.java:724)

Issue Links

• is related to: QPID-4660 [Java Broker] A deadlock can sporadically occur on broker shutdown when stopping a Jetty server which is in the process of sending a response over HTTPS