Description
There are several known OpenJDK JVM bugs (begining with Java11, when TLS v1.3 support was first added) that are known to affect Solr's SSL support, and have caused numerous test failures – notably early "testing" builds of OpenJDK 11, 12, & 13, as well as the officially released OpenJDK 11, 11.0.1, and 11.0.2.
From the standpoint of the Solr project, there is very little we can do to mitigate these bugs, and have taken steps to ensure any code using our SSLTestConfig / RandomizeSSL test-framework classes will be "SKIPed" with an AssumptionViolatedException when used on JVMs that are known to be problematic.
Users who encounter any of the types of failures described below, or developers who encounter test runs that "SKIP" with a message refering to this issue ID, are encouraged to Upgrade their JVM. (or as a last resort: try disabling "TLSv1.3" in your JVM security properties)
Examples of known bugs as they have manifested in Solr tests...
- https://bugs.openjdk.java.net/browse/JDK-8212885
- "TLS 1.3 resumed session does not retain peer certificate chain"
- affects users with checkPeerNames=true in your SSL configuration
- causes 100% failure rate in Solr's TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName
- can result in exceptions for SolrJ users, or in solr cloud server logs when making intra-node requests, with a root cause of "javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated"
[junit4] 2> Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated [junit4] 2> at java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526) [junit4] 2> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:464) [junit4] 2> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397) [junit4] 2> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) [junit4] 2> at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) [junit4] 2> at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) [junit4] 2> at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) [junit4] 2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) [junit4] 2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) [junit4] 2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) [junit4] 2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) [junit4] 2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) [junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) [junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) [junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
- https://bugs.openjdk.java.net/browse/JDK-8213202
- "Possible race condition in TLS 1.3 session resumption"
- May affect any and all Solr SSL users, although noted only in tests when "clientAuth" was configured to be false
- Causes non-reproducing test failures, and sporadic end user exceptions with a root cause of "javax.net.ssl.SSLException: Received fatal alert: internal_error "
- SSL Debugging may indicate "Fatal (INTERNAL_ERROR): Session has no PSK"
[junit4] 2> Caused by: javax.net.ssl.SSLException: Received fatal alert: internal_error [junit4] 2> at sun.security.ssl.Alert.createSSLException(Alert.java:129) ~[?:?] [junit4] 2> at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] [junit4] 2> at sun.security.ssl.TransportContext.fatal(TransportContext.java:308) ~[?:?] [junit4] 2> at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279) ~[?:?] [junit4] 2> at sun.security.ssl.TransportContext.dispatch(TransportContext.java:181) ~[?:?] [junit4] 2> at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?] [junit4] 2> at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ~[?:?] [junit4] 2> at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) ~[?:?] [junit4] 2> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) ~[?:?] [junit4] 2> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6] [junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:555) ~[java/:?] [junit4] 2> ... 13 more
- https://bugs.openjdk.java.net/browse/JDK-8224829
- "AsyncSSLSocketClose.java has timing issue"
- May affect any and all Solr SSL users running early testing versions of java 13 or 14.
- Causes non-reproducing test failures, and sporadic end user exceptions with a root cause of "javax.net.ssl.SSLException: Software caused connection abort: recv failed"
javax.net.ssl.SSLException: Software caused connection abort: recv failed at __randomizedtesting.SeedInfo.seed([AA73C7E858ABD2EE:88D2A395FDC7B4AB]:0) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258) at java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1501) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:935) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
- https://bugs.openjdk.java.net/browse/JDK-8226338
- "Updates to Stateless Resumption"
- May affect any and all Solr SSL servers running early testing or EA builds of java 13 or 14
- Causes reliably reproducing test failures, and Solr server exceptions with a root cause of "java.lang.NullPointerException" in "java.base/sun.security.ssl.SSLSessionImpl.getValue" (or other "Value" related methods in SSLSessionImpl)
java.lang.NullPointerException at java.base/sun.security.ssl.SSLSessionImpl.getValue(SSLSessionImpl.java:1253) at org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:230) at org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:170) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:363) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:144) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.base/java.lang.Thread.run(Thread.java:830)
Attachments
Attachments
Issue Links
- depends upon
-
SOLR-12639 Umbrella JIRA for adding support HTTP/2, jira/http2
-
- Closed
-
- is blocked by
-
SOLR-13574 harden tests that fail (usually NPE) during After/AfterClas methods if there is an assumption violation in Before/BeforeClass methods
-
- Resolved
-
- is duplicated by
-
SOLR-12990 High test failure rate on Java11/12 when (randomized) ssl=true clientAuth=false
-
- Resolved
-
- is related to
-
-
- Resolved
-
-
SOLR-13747
'ant test' should fail on JVM's w/known SSL bugs
-
- Closed
-
- relates to
-
-
- Open
-
(1 relates to)