Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-12265

Upgrade Jetty to 9.4.10

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.4, 8.0
    • Component/s: None
    • Labels:
      None

      Description

      Solr 7.3 upgraded to Jetty 9.4.8 

      We're seeing this WARN very sporadically ( maybe one in every 100k requests ) on the replica when indexing.

      date time WARN [qtp768306356-580185] ? (:) - 
      java.nio.channels.ReadPendingException: null
      at org.eclipse.jetty.io.FillInterest.register(FillInterest.java:58) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.AbstractEndPoint.fillInterested(AbstractEndPoint.java:353) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.AbstractConnection.fillInterested(AbstractConnection.java:134) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:289) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:149) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
      at java.lang.Thread.run(Thread.java:748) [?:1.8.0-zing_17.11.0.0]
      
      date time WARN [qtp768306356-580185] ? (:) - Read pending for org.eclipse.jetty.server.HttpConnection$BlockingReadCallback@2e98df28 prevented AC.ReadCB@424271f8{HttpConnection@424271f8[p=HttpParser{s=START,0 of -1},g=HttpGenerator@424273ae{s=START}]=>HttpChannelOverHttp@4242713d{r=141,c=false,a=IDLE,uri=null}<-DecryptedEndPoint@4242708d{/host:52824<->/host:port,OPEN,fill=FI,flush=-,to=1/86400}->HttpConnection@424271f8[p=HttpParser{s=START,0 of -1},g=HttpGenerator@424273ae{s=START}]=>

      When this happens the leader basically waits till it get's a SocketTimeoutException and then puts the replica into recovery.

      My motivation for upgrading to Jetty 9.4.9 is that the EatWhatYouKill was introduced in Jetty 9.4.x . I don't believe we saw this error in Jetty 9.3.x and then in Jetty 9.4.9 this class has undergone quite a few changes https://github.com/eclipse/jetty.project/commit/0cb4f5629dca082eec943b94ec8ef4ca0d5f1aa4#diff-ae450a12d4eca85a437bd5082f698f48 . 

       

        Attachments

        1. SOLR-12265-shaded-jetty-start.patch
          1 kB
          Steve Rowe
        2. SOLR-12265.patch
          10 kB
          Varun Thacker

          Activity

            People

            • Assignee:
              varunthacker Varun Thacker
              Reporter:
              varunthacker Varun Thacker
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: