ActiveMQ Apollo
  1. ActiveMQ Apollo
  2. APLO-287

SSL errors with Java 7 (Diffie-Hellman cypher suite sessions)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7
    • Component/s: apollo-cli
    • Labels:
      None
    • Environment:
      apollo-99-trunk-20130115.031918-158

      Description

      When running simple functionality tests (that use STOMP+SSL) against apollo-99-trunk-20130115.031918-158, everything works fine when the broker runs on top of a Java 6 JVM.

      However, the exact same tests again the exact same version of Apollo fail randomly when the broker runs on top of a Java 7 JVM. For the record, we use: Java HotSpot(TM) 64-Bit Server VM 1.7.0_10 (Oracle Corporation).

      Here is what the broker logs:

      2013-01-17 11:09:09,663 Shutting connection 'null' down due to: java.io.EOFException: Peer disconnected
      2013-01-17 11:09:09,764 Shutting connection 'null' down due to: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
      2013-01-17 11:11:16,110 Shutting connection 'null' down due to: javax.net.ssl.SSLHandshakeException: Invalid padding
      2013-01-17 11:13:54,671 Shutting connection 'null' down due to: javax.net.ssl.SSLHandshakeException: Invalid padding
      [...]

      First of all, is Apollo supported on Java 7?

      If yes, any idea what could cause these failures on Java 7?

        Activity

        Lionel Cons created issue -
        Hide
        Lionel Cons added a comment -

        FWIW, I got the same errors with apollo-99-trunk-20130119.032027-161.

        Also, others got similar errors with other applications: http://community.igniterealtime.org/thread/48499.

        BTW, it would probably help to log a stack trace for these exceptions.

        Show
        Lionel Cons added a comment - FWIW, I got the same errors with apollo-99-trunk-20130119.032027-161. Also, others got similar errors with other applications: http://community.igniterealtime.org/thread/48499 . BTW, it would probably help to log a stack trace for these exceptions.
        Hide
        Hiram Chirino added a comment -

        Looks related to http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6946669
        Need to figure out which JDK 7 release the fix went into.

        Show
        Hiram Chirino added a comment - Looks related to http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6946669 Need to figure out which JDK 7 release the fix went into.
        Hide
        Hiram Chirino added a comment -

        Do you have any scripts that can reproduce?

        Show
        Hiram Chirino added a comment - Do you have any scripts that can reproduce?
        Hide
        Hiram Chirino added a comment -

        BTW, I'm using Java 7 as my primary development platform, so yes Java 7 is supported.

        Show
        Hiram Chirino added a comment - BTW, I'm using Java 7 as my primary development platform, so yes Java 7 is supported.
        Hide
        Hiram Chirino added a comment -

        Closing as I cannot reproduce the error. Please re-open if you figure out how to reproduce.

        Show
        Hiram Chirino added a comment - Closing as I cannot reproduce the error. Please re-open if you figure out how to reproduce.
        Hiram Chirino made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee Hiram Chirino [ chirino ]
        Resolution Cannot Reproduce [ 5 ]
        Hide
        Hiram Chirino added a comment -

        Seems like lots of folks are runnign into issues with SSL on Java 1.7. See: https://forums.oracle.com/forums/thread.jspa?messageID=10875177&tstart=0&utm_source=buffer&buffer_share=54d10

        Show
        Hiram Chirino added a comment - Seems like lots of folks are runnign into issues with SSL on Java 1.7. See: https://forums.oracle.com/forums/thread.jspa?messageID=10875177&tstart=0&utm_source=buffer&buffer_share=54d10
        Hiram Chirino made changes -
        Resolution Cannot Reproduce [ 5 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Hiram Chirino added a comment - - edited

        Folks in the post in the previous thread reported that disabling the Diffie-Hellman cypher suite lets them work around the issue /w Java 7. I've just deployed a new build of Apollo which supports disabling the Diffie-Hellman cypher suite.

        Lionel, if you get a chance could you try out the new snapshot build on Java 7 and update the connector configuration similar to the following:

        <connector id="ssl" bind="ssl://0.0.0.0:61614?disabled_cypher_suites=DHE" />

        Let me know if disabling the DHE cyphers helps out on Java 7.

        Show
        Hiram Chirino added a comment - - edited Folks in the post in the previous thread reported that disabling the Diffie-Hellman cypher suite lets them work around the issue /w Java 7. I've just deployed a new build of Apollo which supports disabling the Diffie-Hellman cypher suite. Lionel, if you get a chance could you try out the new snapshot build on Java 7 and update the connector configuration similar to the following: <connector id="ssl" bind="ssl://0.0.0.0:61614?disabled_cypher_suites=DHE" /> Let me know if disabling the DHE cyphers helps out on Java 7.
        Hide
        Lionel Cons added a comment -

        Unfortunately, it seems that we use Diffie-Hellman since, when disabling it, my client fails to authenticate and gets:

        Authentication failed. Credentials=[]

        Show
        Lionel Cons added a comment - Unfortunately, it seems that we use Diffie-Hellman since, when disabling it, my client fails to authenticate and gets: Authentication failed. Credentials=[]
        Hide
        Hiram Chirino added a comment -

        Hi, could you enable the JVM's SSL debug logging by adding the '-Djavax.net.debug=all' JVM option? Typically that can be done by setting the following env variable before running Apollo: APOLLO_OPTS=-Djavax.net.debug=all

        The SSL debug data will be sent to the console. That should give us more details what cypher suites are being picked etc. Also what OS/SSL library is your client using. Is there something you can share to reproduce?

        Show
        Hiram Chirino added a comment - Hi, could you enable the JVM's SSL debug logging by adding the '-Djavax.net.debug=all' JVM option? Typically that can be done by setting the following env variable before running Apollo: APOLLO_OPTS=-Djavax.net.debug=all The SSL debug data will be sent to the console. That should give us more details what cypher suites are being picked etc. Also what OS/SSL library is your client using. Is there something you can share to reproduce?
        Hide
        Gary Tully added a comment -

        u may have to create new certs that don't use DH in the signatures to workaround.

        Show
        Gary Tully added a comment - u may have to create new certs that don't use DH in the signatures to workaround.
        Hide
        Lionel Cons added a comment -

        Gary, we use certificates coming from tens of CAs around the world so changing the way they are created is unfortunately not an option.

        Show
        Lionel Cons added a comment - Gary, we use certificates coming from tens of CAs around the world so changing the way they are created is unfortunately not an option.
        Hide
        Lionel Cons added a comment -

        Hiram, I've enabled debugging but this is very verbose: I logged >100MB in only a few minutes. Also, the broker seemed slowed down because of this logging and the test results were different.

        Most sessions appeared as:

        %% Initialized: [Session-3, SSL_NULL_WITH_NULL_NULL]
        %% Negotiating: [Session-3, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA]
        ...
        Cipher Suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        Compression Method: 0
        Extension renegotiation_info, renegotiated_connection: <empty>

        Regarding the client used, this is using Perl + Net::STOMP::Client.

        You should be able to perform the same tests by fetching the same package (http://svn.cern.ch/guest/MIG/trunk/mbtf) and follow the instructions in README.howto.

        Show
        Lionel Cons added a comment - Hiram, I've enabled debugging but this is very verbose: I logged >100MB in only a few minutes. Also, the broker seemed slowed down because of this logging and the test results were different. Most sessions appeared as: %% Initialized: [Session-3, SSL_NULL_WITH_NULL_NULL] %% Negotiating: [Session-3, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA] ... Cipher Suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA Compression Method: 0 Extension renegotiation_info, renegotiated_connection: <empty> Regarding the client used, this is using Perl + Net::STOMP::Client. You should be able to perform the same tests by fetching the same package ( http://svn.cern.ch/guest/MIG/trunk/mbtf ) and follow the instructions in README.howto.
        Hide
        Lionel Cons added a comment -

        After a few more attempts and 1.5GB of logs, I've reproduced the "invalid padding" error and got:

        hawtdispatch-DEFAULT-1, fatal error: 40: Invalid Padding length: 158
        javax.crypto.BadPaddingException: Invalid Padding length: 158
        %% Invalidated: [Session-257, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA]
        hawtdispatch-DEFAULT-1, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: Invalid Padding length: 158

        So this indeed looks a lot like the OTN link you gave. If indeed the problem is on the Java side, how can we help bug fixing on the Oracle side?

        Show
        Lionel Cons added a comment - After a few more attempts and 1.5GB of logs, I've reproduced the "invalid padding" error and got: hawtdispatch-DEFAULT-1, fatal error: 40: Invalid Padding length: 158 javax.crypto.BadPaddingException: Invalid Padding length: 158 %% Invalidated: [Session-257, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA] hawtdispatch-DEFAULT-1, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: Invalid Padding length: 158 So this indeed looks a lot like the OTN link you gave. If indeed the problem is on the Java side, how can we help bug fixing on the Oracle side?
        Hide
        Hiram Chirino added a comment -

        Ok, lets try something else. Luckily the crypto implementations in the JVM are pluggable, and the bouncycastle folks have a very excellent implementation. I've just deployed a new version of Apollo [1] which will install the bouncycastle security provider if it finds it in the classpath. You just need to install the bcprov-jdk15on-148.jar [2] in Apollo's lib directory. You'll know it got loaded if on startup, you see the 'Loaded the Bouncy Castle security provider.' message logged to the console.

        [1]: https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-apollo/99-trunk-SNAPSHOT/apache-apollo-99-trunk-20130405.150934-220-unix-distro.tar.gz
        [2]: http://www.bouncycastle.org/download/bcprov-jdk15on-148.jar

        Lionel, could you check to see if bouncycastle impl behaves better than the default JVM impl.

        Show
        Hiram Chirino added a comment - Ok, lets try something else. Luckily the crypto implementations in the JVM are pluggable, and the bouncycastle folks have a very excellent implementation. I've just deployed a new version of Apollo [1] which will install the bouncycastle security provider if it finds it in the classpath. You just need to install the bcprov-jdk15on-148.jar [2] in Apollo's lib directory. You'll know it got loaded if on startup, you see the 'Loaded the Bouncy Castle security provider.' message logged to the console. [1] : https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-apollo/99-trunk-SNAPSHOT/apache-apollo-99-trunk-20130405.150934-220-unix-distro.tar.gz [2] : http://www.bouncycastle.org/download/bcprov-jdk15on-148.jar Lionel, could you check to see if bouncycastle impl behaves better than the default JVM impl.
        Hide
        Lionel Cons added a comment -

        Hiram, I ran several tests and I confirm that the problem goes away when using the Bouncy Castle jar. Thanks for the workaround while the Java bug gets fixed.

        However, since console is rarely used (we start Apollo via as a service), it would be better to log the provider info as a normal message ending up in apollo.log...

        Show
        Lionel Cons added a comment - Hiram, I ran several tests and I confirm that the problem goes away when using the Bouncy Castle jar. Thanks for the workaround while the Java bug gets fixed. However, since console is rarely used (we start Apollo via as a service), it would be better to log the provider info as a normal message ending up in apollo.log...
        Hiram Chirino made changes -
        Summary Weird errors with SSL + Java 7 SSL errors with Java 7 (Diffie-Hellman cypher suite sessions)
        Hide
        Hiram Chirino added a comment -

        Thanks Lionel! I've made that change and I've updated the docs to explain the issue and workaround.

        Show
        Hiram Chirino added a comment - Thanks Lionel! I've made that change and I've updated the docs to explain the issue and workaround.
        Hiram Chirino made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hiram Chirino made changes -
        Fix Version/s 1.7 [ 12322515 ]
        Hiram Chirino made changes -
        Component/s apollo-cli [ 12314343 ]
        Hide
        Lionel Cons added a comment -

        For the record, here is a pointer to the underlying Java bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8013059

        Show
        Lionel Cons added a comment - For the record, here is a pointer to the underlying Java bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8013059

          People

          • Assignee:
            Hiram Chirino
            Reporter:
            Lionel Cons
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development