[AMQ-4312] NIO+SSL Connector fails with SSL exception under high concurrency

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.7.0, 5.8.0
• Fix Version/s: 5.9.0
• Component/s: Transport
• Labels: nio ssl transport

Environment

JDK 1.7.0_09, Windows 7 x64, Linux CentOS5 64-bit

• Patch Info: Patch Available
• Assignee: Timothy A. Bish
• Reporter: Craig Condit
• Created: 12/Feb/13 22:42
• Updated: 13/Feb/13 00:41
• Resolved: 13/Feb/13 00:40

Description

Under high concurrency, the NIO+SSL connector causes client connections to fail with random SSL exceptions (usually bad record MAC or invalid padding) after a period of time.

For example:

javax.net.ssl.SSLException: bad record MAC
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1855)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:988)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
	at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:604)
	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
	at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:589)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
	at java.lang.Thread.run(Thread.java:722)</pre>

I suspect this may be related to AMQ-4135, since we are seeing nearly identical failure modes.

I am able to duplicate these results fairly consistently using a variant of NIOSSLLoadTest which uses 10 producers, 10 consumers, 10000 messages, and a bytes message of size 4096 with all producers and consumers using separate (multiplexed) sessions.

Issue Links

• is related to: AMQ-4135 Bug in stomp+nio+ssl