[OAK-6500] NRTIndex leaks file handles due to unclosed IndexReader

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.6, 1.6.4, 1.8.0
• Component/s: lucene
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 27/Jul/17 07:40
• Updated: 25/Oct/17 13:17
• Resolved: 04/Aug/17 06:01

Description

On some setups under stress it has been seen that NRTIndex leaks file handles over time.

Checking with lsof indicates that more than 3 nrt folders per index are being used. However per design there can be max 3 and after system is not in use max 1 should be present.

$ lsof -p 9550 | grep '/nrt' | gawk 'match($0, /.*crx-quickstart\/repository\/index\/(.*?)\/\_.*$/, m) { print m[1]; }' | sort | uniq
cqPageLucene-1501065263331/nrt1501065335930
cqPageLucene-1501065263331/nrt1501065374667
cqPageLucene-1501065263331/nrt1501065392492
cqPageLucene-1501065263331/nrt1501065440955
cqPageLucene-1501065263331/nrt1501065473286
cqPageLucene-1501065263331/nrt1501065507345
slingeventJob-1501065263330/nrt1501065356975
slingeventJob-1501065263330/nrt1501065373229
slingeventJob-1501065263330/nrt1501065394142
slingeventJob-1501065263330/nrt1501065440953
slingeventJob-1501065263330/nrt1501065473282
slingeventJob-1501065263330/nrt1501065507342
versionStoreIndex-1501065263332/nrt1501065335925
versionStoreIndex-1501065263332/nrt1501065366781
versionStoreIndex-1501065263332/nrt1501065392490
versionStoreIndex-1501065263332/nrt1501065441232
versionStoreIndex-1501065263332/nrt1501065473285

Further actually checking index folder indicates that those folder are actually deleted. So some where the file handle is still referring them.

Issue Links

• breaks: OAK-6572 IndexReaderClosed exception seen after some run
• breaks: OAK-6635 IndexReader closed exception in DocumentQueue
• relates to: OAK-6635 IndexReader closed exception in DocumentQueue