ActiveMQ
  1. ActiveMQ
  2. AMQ-4312

NIO+SSL Connector fails with SSL exception under high concurrency

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.7.0, 5.8.0
    • Fix Version/s: 5.9.0
    • Component/s: Transport
    • Labels:
    • Environment:

      JDK 1.7.0_09, Windows 7 x64, Linux CentOS5 64-bit

    • Patch Info:
      Patch Available

      Description

      Under high concurrency, the NIO+SSL connector causes client connections to fail with random SSL exceptions (usually bad record MAC or invalid padding) after a period of time.

      For example:

      javax.net.ssl.SSLException: bad record MAC
      	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
      	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
      	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1855)
      	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:988)
      	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
      	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
      	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
      	at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:604)
      	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
      	at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:589)
      	at java.io.DataInputStream.readInt(DataInputStream.java:387)
      	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
      	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
      	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
      	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
      	at java.lang.Thread.run(Thread.java:722)</pre>
      

      I suspect this may be related to AMQ-4135, since we are seeing nearly identical failure modes.

      I am able to duplicate these results fairly consistently using a variant of NIOSSLLoadTest which uses 10 producers, 10 consumers, 10000 messages, and a bytes message of size 4096 with all producers and consumers using separate (multiplexed) sessions.

      1. AMQ-4312-test.patch
        9 kB
        Craig Condit
      2. AMQ-4312.patch
        0.8 kB
        Craig Condit
      3. NIOSSLConcurrencyTest.java
        9 kB
        Craig Condit

        Issue Links

          Activity

          Hide
          Timothy Bish added a comment -

          Patch applied, great work.

          Show
          Timothy Bish added a comment - Patch applied, great work.
          Hide
          Timothy Bish added a comment -

          Nice find, looks right. I'm testing things now for any side effects.

          Show
          Timothy Bish added a comment - Nice find, looks right. I'm testing things now for any side effects.
          Hide
          Craig Condit added a comment -

          Attached patch for unit test.

          Show
          Craig Condit added a comment - Attached patch for unit test.
          Hide
          Craig Condit added a comment -

          Attached patch which seems to fix the issue.

          Show
          Craig Condit added a comment - Attached patch which seems to fix the issue.
          Hide
          Craig Condit added a comment - - edited

          Attached unit test which demonstrates the problem.

          Show
          Craig Condit added a comment - - edited Attached unit test which demonstrates the problem.

            People

            • Assignee:
              Timothy Bish
              Reporter:
              Craig Condit
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development