[SOLR-1911] File descriptor leak while indexing, may cause index corruption

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 1.5
• Component/s: update

Environment

Ubuntu Linux, Java build 1.6.0_16-b01
Solr Specification Version: 3.0.0.2010.05.12.16.17.46
Solr Implementation Version: 4.0-dev exported - simon - 2010-05-12 16:17:46 – bult from updated trunk
Lucene Specification Version: 4.0-dev
Lucene Implementation Version: 4.0-dev exported - 2010-05-12 16:18:26
Current Time: Thu May 13 12:21:12 EDT 2010
Server Start Time:Thu May 13 11:45:41 EDT 2010

• Assignee: Unassigned
• Reporter: Simon Rosenthal
• Created: 13/May/10 16:57
• Updated: 16/Mar/11 16:59
• Resolved: 16/Mar/11 16:59

Description

While adding documents to an already existing index using this build, the number of open file descriptors increases dramatically until the open file per-process limit is reached (1024) , at which point there are error messages in the log to that effect. If the server is restarted the index may be corrupt

commits are handled by autocommit every 60 seconds or 500 documents (usually the time limit is reached first).
mergeFactor is 10.

It looks as though each time a commit takes place, the number of open files (obtained from " lsof -p `cat solr.pid` | egrep ' [0-9]+r ' ") increases by 40, There are several open file descriptors associated with each file in the index.

Rerunning the same index updates with an older Solr (built from trunk in Feb 2010) doesn't show this problem - the number of open files fluctuates up and down as segments are created and merged, but stays basically constant.