Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14896

jetty "Bad Message 400" / "Illegal character" responses to sporadic requests



    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 8.6.3, 8.7, 9.0
    • None
    • None


      I'm getting some offline reports from users who recently upgraded to solr 8.6 of non-reliably reproducing errors from Jetty when doing the HTTP header parsing. these errors result in a jetty response that will typically look something like...

      HTTP/1.1 400 Illegal character VCHAR='='
      Content-Type: text/html;charset=iso-8859-1
      Content-Length: 70
      Connection: close
      <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre>

      ...the exact "Illegal character" can vary. Based on some review of the solr/jetty code bases these errors come from Jetty's HttpParser class when unexpected character types are encountered while parsing the HTTP METHOD or Header NAMES (VCHAR is a legal token type in Header VALUES)

      These errors sometimes manifest in log files as RemoteSolrExceptions due to intra-node communication, but I've been told they can also be returned directly by solr/jetty when clients make requests – even when the client is not using SolrJ – suggesting that whatever is causing the "malformed" request is not specifically a bug in solr/solrj – this seems like a jetty bug.

      I'm suspect that the underlying cause is jetty buffers being re-used/shared between multiple requests, and suspicious that this is caused by / related to jetty issue#4936 / aka jetty bug#564984 .. BUT ... that issue suggests that the buffer re-use situation ONLY arises due to a prior "500: Response header too large" error – but i have not been able to confirm that that situation has ever existed on any of the affected servers where this "400 Illegal character" error occurs.


        Issue Links



              Unassigned Unassigned
              hossman Chris M. Hostetter
              0 Vote for this issue
              3 Start watching this issue