The machine has 4GB total. In response to this issue, and especially now that we have upgraded it to be 64 bit (again, for this issue), we have already ordered another 16 GB for the machine to try and stave off the problem. We should have it in next week.
I restrict commits severely - a commit is only allowed once an hour, in practice they happen even less frequently - perhaps 5 or 6 times a day, and very spread out. We are freakishly paranoid But honestly that's all we need - new documents come in in chunks and generally they want them to go in all at once, and not piecemeal, so that the site updates cleanly (the commits are synchronized with other content updates - new images on the home page, etc).
Some more information... just trying to toss out anything that matters. We have a very small set of possible terms - only 60,000 or so which tokenize to perhaps 200,000 total distinct words. We do not use synonyms at index time (only at query time). We use faceting, collapsing, and sorting - that's about it, no more like this or spellchecker (although we'd like to, we haven't gotten there yet). Faceting we do use heavily though - there are 16 different fields on which we return facet counts. All these fields together represent no more than 15,000 unique terms. There are approx. 4M documents in the index total, and none of them are larger than 1K.
Memory usage on the machine seems to steadily increase - after restart and warming, 40% of the RAM on the machine is in use. Then, as searches come in, it steadily increases. Right now it is using 61%, in an hour it will probably be closer to 75% - the danger zone. This is also unusual because before, it used to stay pretty steady around 52-53%.
This is a multi-core system - we have 2 cores, the one I'm describing now is only one of them. The other core is very, very small - total 8000 documents, which are also no more than 1 K each. We do use faceting there but no collapsing (it is not necessary for that part). It is essentially irrelevant, with or without that core the machine consumes about the same amount of resources.
In response to this problem I have already dramatically reduced the following options:
( diff of solrconfig.xml - < indicates current values, > indicates values when the problem started happening).
This actually seemed to make the search much faster (strangely enough), but it doesn't seem to have helped memory consumption very much.
These are our cache parameters:
I'm actually not sure if the collapseCache even does anything since it does not appear in the admin listing. I'm going to try reducing the filterCache to 32K entries and see if that makes a difference. I think that may be the right track since otherwise it seems like a big memory leak is happening.
Is there any way to specify the size of the cache in terms of the actual size it should take up in memory, as opposed to the number of entries? 64K sounded quite small to me but now I'm thinking that 64K could mean GB's of memory depending on what the entries are, I honestly don't understand what the correlation would be between an entry and the size that entry takes in RAM.