Major Description
Solr 7.3 upgraded to Jetty 9.4.8
We're seeing this WARN very sporadically ( maybe one in every 100k requests ) on the replica when indexing.
date time WARN [qtp768306356-580185] ? (:) - java.nio.channels.ReadPendingException: null at org.eclipse.jetty.io.FillInterest.register(FillInterest.java:58) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.AbstractEndPoint.fillInterested(AbstractEndPoint.java:353) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.AbstractConnection.fillInterested(AbstractConnection.java:134) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:289) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:149) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121] at java.lang.Thread.run(Thread.java:748) [?:1.8.0-zing_17.11.0.0] date time WARN [qtp768306356-580185] ? (:) - Read pending for org.eclipse.jetty.server.HttpConnection$BlockingReadCallback@2e98df28 prevented AC.ReadCB@424271f8{HttpConnection@424271f8[p=HttpParser{s=START,0 of -1},g=HttpGenerator@424273ae{s=START}]=>HttpChannelOverHttp@4242713d{r=141,c=false,a=IDLE,uri=null}<-DecryptedEndPoint@4242708d{/host:52824<->/host:port,OPEN,fill=FI,flush=-,to=1/86400}->HttpConnection@424271f8[p=HttpParser{s=START,0 of -1},g=HttpGenerator@424273ae{s=START}]=>
When this happens the leader basically waits till it get's a SocketTimeoutException and then puts the replica into recovery.
My motivation for upgrading to Jetty 9.4.9 is that the EatWhatYouKill was introduced in Jetty 9.4.x . I don't believe we saw this error in Jetty 9.3.x and then in Jetty 9.4.9 this class has undergone quite a few changes https://github.com/eclipse/jetty.project/commit/0cb4f5629dca082eec943b94ec8ef4ca0d5f1aa4#diff-ae450a12d4eca85a437bd5082f698f48 .