Details
Major Description
We have a set of standalone solr nodes running on Solr 7.5. We recently had a few episodes where the entire cluster crashed and died all together. Digging in a little, we found the culprits were some SQL injection attacks happening on our site where the search term had SQL injection in it and that was fed into the q param in solr. I was able to take a stable solr and isolate it and just run 1 query and make it crash. Every time I would run a regular query and see it work and then just change the q= parameter and that would time out and eventually crash the solr instance. Here is the q param for the query I ran:
q=6792)))+UNION+ALL+SELECT+NULL,NULL,NULL,NULL,CHR(113)||CHR(98)||CHR(118)||CHR(113)||CHR(113)||CHR(104)||CHR(68)||CHR(86)||CHR(114)||CHR(109)||CHR(97)||CHR(89)||CHR(89)||CHR(112)||CHR(76)||CHR(90)||CHR(105)||CHR(113)||CHR(86)||CHR(102)||CHR(97)||CHR(108)||CHR(89)||CHR(83)||CHR(81)||CHR(107)||CHR(69)||CHR(111)||CHR(97)||CHR(75)||CHR(87)||CHR(68)||CHR(108)||CHR(73)||CHR(68)||CHR(86)||CHR(118)||CHR(101)||CHR(71)||CHR(78)||CHR(106)||CHR(106)||CHR(76)||CHR(65)||CHR(82)||CHR(113)||CHR(106)||CHR(98)||CHR(98)||CHR(113)+FROM+DUAL-+gKiW
I even stripped out the "||" characters and replaced them with "," and it still crashes. Please note these were SQL injection attacks and not real good queries. The Solr GC log exposes the problem and shows the memory footprint ballooning (from 2GB to 18GB within a minute) to the point where full garbage collection fails and the Solr instance is unresponsive. So 1 query is able to push it to the tipping point and consume 18GB of memory.
I have tried searching for long description texts but that works fine. So something with these characters is probably causing this. Does anyone know how/why this might be happening?