[OAK-5661] Make NRT indexing resilient against unbounded growth

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: lucene
• Epic Link: indexer resilience
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 15/Feb/17 04:23
• Updated: 26/May/21 14:52

Description

NRT Indexes for volatile indexes [1] can grow large if async index update faces issues. Like if it gets stuck for days or due to some bug index do not get updates like in OAK-5649 then the sizes can grow very large.

For such cases we should add some checks in logic where system can ensure that some cleanup is performed or writes to indexes are stopped. Also such a situation should be flagged

[1] Indexes which see lots of addition and deletions. So effective indexing size is smaller however if deletions are not applied (as is the case with NRT) such indexes can grow large

Issue Links

• relates to: OAK-5649 Error in RefreshPolicy can lead to IndexNode lock leak