[SOLR-14896] jetty "Bad Message 400" / "Illegal character" responses to sporadic requests - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 8.6.3, 8.7, 9.0
Component/s: None
Labels:
None

Description

I'm getting some offline reports from users who recently upgraded to solr 8.6 of non-reliably reproducing errors from Jetty when doing the HTTP header parsing. these errors result in a jetty response that will typically look something like...

HTTP/1.1 400 Illegal character VCHAR='='
Content-Type: text/html;charset=iso-8859-1
Content-Length: 70
Connection: close

<h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='='</pre>

...the exact "Illegal character" can vary. Based on some review of the solr/jetty code bases these errors come from Jetty's HttpParser class when unexpected character types are encountered while parsing the HTTP METHOD or Header NAMES (VCHAR is a legal token type in Header VALUES)

These errors sometimes manifest in log files as RemoteSolrExceptions due to intra-node communication, but I've been told they can also be returned directly by solr/jetty when clients make requests – even when the client is not using SolrJ – suggesting that whatever is causing the "malformed" request is not specifically a bug in solr/solrj – this seems like a jetty bug.

I'm suspect that the underlying cause is jetty buffers being re-used/shared between multiple requests, and suspicious that this is caused by / related to jetty issue#4936 / aka jetty bug#564984 .. BUT ... that issue suggests that the buffer re-use situation ONLY arises due to a prior "500: Response header too large" error – but i have not been able to confirm that that situation has ever existed on any of the affected servers where this "400 Illegal character" error occurs.