[SOLR-16084] When Solr server runs with HTTPS java clients can't connect but curl can.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: clients - java

Environment

java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment Temurin-17.0.2+8 (build 17.0.2+8)
OpenJDK 64-Bit Server VM Temurin-17.0.2+8 (build 17.0.2+8, mixed mode, sharing)

The application is using spring boot 2.5.6, solr-solrj 8.11.1, and apache httpclient 4.5.13. It is also forcing TLS 1.2. 

• Assignee: Unassigned
• Reporter: Peter Lebedev
• Created: 07/Mar/22 21:05
• Updated: 12/May/22 00:35

Description

When we have Solr server running on HTTPS, Http2SolrClient is not able to communicate with it, while curl works, and apache HTTP client, org.apache.http.impl.client.CloseableHttpClient, if called from the same Java application.

Here is the application level exception:

Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://jobcase-solr-job-prod-replica.sensitive-marketplace-staging.ojop.io/solr
78	at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:437) ~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
79	at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776) ~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
80	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214) ~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
89	... 61 more
90Caused by: java.io.IOException: Broken pipe
91	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1135) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
92	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
93	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
94	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
95	at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:295) ~[http2-common-9.4.44.v20210927.jar!/:9.4.44.v20210927]
96	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) ~[jetty-util-9.4.44.v20210927.jar!/:9.4.44.v20210927]
97	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223) ~[jetty-util-9.4.44.v20210927.jar!/:9.4.44.v20210927]
98	at org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:729) ~[http2-common-9.4.44.v20210927.jar!/:9.4.44.v20210927]
99	at org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:701) ~[http2-common-9.4.44.v20210927.jar!/:9.4.44.v20210927]
100	at org.eclipse.jetty.http2.client.HTTP2ClientConnectionFactory$HTTP2ClientConnection.onOpen(HTTP2ClientConnectionFactory.java:116) ~[http2-client-9.4.44.v20210927.jar!/:9.4.44.v20210927]
101	at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:444) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
102	at org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:114) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
103	at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:84) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
104	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
105	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
106	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
107	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
108	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
109	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
110	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.44.v20210927.jar!/:9.4.44.v20210927]
111	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218) ~[solr-solrj-8.11.1.jar!/:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:57]
112	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
113	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?] 

When I enable ssl debug, I see this in the logs:

javax.net.ssl|ERROR|7B|h2sc-1-thread-5|2022-03-04 15:15:22.809 EST|TransportContext.java:363|Fatal (INTERNAL_ERROR): closing inbound before receiving peer's close_notify (
546"throwable" : {
547  javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
548  	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
549  	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
550  	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:358)
551  	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
552  	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:305)
553  	at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:796)
554  	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.closeInbound(SslConnection.java:982)
555  	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:766)
556  	at org.eclipse.jetty.io.NegotiatingClientConnection.fill(NegotiatingClientConnection.java:99)
557  	at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:81)
558  	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
559  	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
560  	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
561  	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
562  	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
563  	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
564  	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
565  	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
566  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
567  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
568  	at java.base/java.lang.Thread.run(Thread.java:833)} 

This seems to be related to https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8253368 

How can I make it working with https Solr server?
It is possible that the issue is with jetty HTTP client, not with Solr client directly, so it could be a misconfiguration. However, there is no way to provide externally configured jetty http client, nor it supports the same level of configuration as apache http client. For example, when I build apache HttpClient I set these on its builder:

builder.setSSLHostnameVerifier(hostnameVerifier);
builder.setSSLContext(sslContext);
builder.setSSLSocketFactory(sslConnSocketFactory); 

and https connections work to the same Solr server.