Details
-
Improvement
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
main (10.0), 9.4
-
None
-
None
Description
As mkhl suggested on SOLR-13315, it would be possible for LogWatcher, rather than buffering all the raw parameters of a log message, to pre-compute the derivative {{SolrDocument}}s and buffer those instead.
SOLR-13315 ended up being addressed in a different way, but "memory-leak-like" behavior is still possible and I think we should follow mkhl's suggestion. The problem we ran into recently was a huge distributed query, dispersed to lots of shards. One of the ShardRequests failed, causing the failed ShardResponse to be logged at warn level in SearchHandler:
log.warn("Shard request failed : {}", srsp);
The logged ShardResponse indirectly holds a reference (via ShardRequest ref) to all the other ShardResponses, some of which succeeded and were quite large. These references then just sit in LogWatcher's history CircularBuffer until they are overwritten by subsequent log messages (default WARN/ERROR level). So as long as you have fewer than 50 (default config) subsequent WARN/ERROR level messages, these references will be held (in principle forever). In our case it consumed a substantial portion of the heap.
This issue proposes that LogWatcher should process LogEvents into SolrDocuments (the form they take upon retrieval) when they are initially added to the history buffer. The downside is some minimal amount of extra processing in the case where the events are never retrieved from the buffer; the upside is that the params don't needlessly retain heap.
Attachments
Issue Links
- links to
