Description
I'm getting some offline reports from users who recently upgraded to solr 8.6 of non-reliably reproducing errors from Jetty when doing the HTTP header parsing. these errors result in a jetty response that will typically look something like...
HTTP/1.1 400 Illegal character VCHAR='=' Content-Type: text/html;charset=iso-8859-1 Content-Length: 70 Connection: close <h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre>
...the exact "Illegal character" can vary. Based on some review of the solr/jetty code bases these errors come from Jetty's HttpParser class when unexpected character types are encountered while parsing the HTTP METHOD or Header NAMES (VCHAR is a legal token type in Header VALUES)
These errors sometimes manifest in log files as RemoteSolrExceptions due to intra-node communication, but I've been told they can also be returned directly by solr/jetty when clients make requests – even when the client is not using SolrJ – suggesting that whatever is causing the "malformed" request is not specifically a bug in solr/solrj – this seems like a jetty bug.
I'm suspect that the underlying cause is jetty buffers being re-used/shared between multiple requests, and suspicious that this is caused by / related to jetty issue#4936 / aka jetty bug#564984 .. BUT ... that issue suggests that the buffer re-use situation ONLY arises due to a prior "500: Response header too large" error – but i have not been able to confirm that that situation has ever existed on any of the affected servers where this "400 Illegal character" error occurs.
Attachments
Issue Links
- is fixed by
-
SOLR-14897 HttpSolrCall will forward a virtually unlimited number of times until ClusterState ZkWatcher is updated after collection delete
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
SOLR-14844 Upgrade Jetty to 9.4.32.v20200930
-
- Resolved
-