Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14896

jetty "Bad Message 400" / "Illegal character" responses to sporadic requests

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.6.3, 8.7, main (9.0)
    • Component/s: None
    • Labels:
      None

      Description

      I'm getting some offline reports from users who recently upgraded to solr 8.6 of non-reliably reproducing errors from Jetty when doing the HTTP header parsing. these errors result in a jetty response that will typically look something like...

      HTTP/1.1 400 Illegal character VCHAR='='
      Content-Type: text/html;charset=iso-8859-1
      Content-Length: 70
      Connection: close
      
      <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre>
      

      ...the exact "Illegal character" can vary. Based on some review of the solr/jetty code bases these errors come from Jetty's HttpParser class when unexpected character types are encountered while parsing the HTTP METHOD or Header NAMES (VCHAR is a legal token type in Header VALUES)


      These errors sometimes manifest in log files as RemoteSolrExceptions due to intra-node communication, but I've been told they can also be returned directly by solr/jetty when clients make requests – even when the client is not using SolrJ – suggesting that whatever is causing the "malformed" request is not specifically a bug in solr/solrj – this seems like a jetty bug.

      I'm suspect that the underlying cause is jetty buffers being re-used/shared between multiple requests, and suspicious that this is caused by / related to jetty issue#4936 / aka jetty bug#564984 .. BUT ... that issue suggests that the buffer re-use situation ONLY arises due to a prior "500: Response header too large" error – but i have not been able to confirm that that situation has ever existed on any of the affected servers where this "400 Illegal character" error occurs.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hossman Chris M. Hostetter
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: