After a code review, one source leak is in ElectionContext.java:

      if (!success && ulog.getRecentUpdates().getVersions(1).isEmpty()) {

introduced in SOLR-3933 (Solr 4.1)

It's actually SOLR-3939

This one actually occured to me when i was reading the user thread on this the other day - it didn't seem like the culprit for that guy though because it only happens on election (unless he was losing the leader consistently for some ugly reason).

Here's a patch that should hopefully fix things up wrt getRecentUpdates.

This looks like it reintroduces the NPE you can get with no ulog in ElectionContext - when I put in the null check yesterday or the day before, I was torn between just letting the node become leader if it has no ulog and was active and throwing a specific exception about having no ulog - i ended up choosing the former thinking if we didn't want to support no ulog in solrcloud mode, that should be checked on startup.

This looks like it reintroduces the NPE you can get with no ulog in ElectionContext

Ah, thanks - I got a merge conflict and then missed your update. I'll fix.

Here's another version that cleans up tlog references during tlog recovery in the event an unexpected exception (like a commit throwing something other than IOException).

Aside: we should keep in mind that when index corruption happens, lucene can throw exceptions other than IOException.

I patched my 4.3.0 install with the attached patch on a solr instance with my broken index that was causing this issue to begin with and i see that the tlog files are limited to 10 files properly and I am not building up orphaned FileDescriptors any more. Additionally I've verified with lsof that the tlog isn't leaking open file...

Thanks for verifying Steven!
Committed to trunk, 4x, 4.3.1

Bulk close after 4.3.1 release