I hit this while testing various use cases for
LUCENE-6119 (adding auto-throttle to ConcurrentMergeScheduler).
When I tested "always call updateDocument" (each add buffers a delete term), with many indexing threads, opening an NRT reader once per second (forcing all deleted terms to be applied), I see that BufferedUpdatesStream.applyDeletes sometimes seems to take a loooong time, e.g.:
What this means is even though I want an NRT reader every second, often I don't get one for up to ~7 or more seconds.
This is on an SSD, machine has 48 GB RAM, heap size is only 2 GB. 12 indexing threads.
As hideously complex as this code is, I think there are some inefficiencies, but fixing them could be hard / make code even hairier ...
Also, this code is mega-locked: holds IW's lock, holds BD's lock. It blocks things like merges kicking off or finishing...
E.g., we pull the MergedIterator many times on the same set of sub-iterators. Maybe we can create the sorted terms up front and reuse that?
Maybe we should go "term stride" (one term visits all N segments) not "segment stride" (visit each segment, iterating all deleted terms for it). Just iterating the terms to be deleted takes a sizable part of the time, and we now do that once for every segment in the index.
Also, the "isUnique" bit in LUCENE-6005 should help here, since if we know the field is unique, we can stop seekExact once we found a segment that has the deleted term, we can maybe pass false for removeDuplicates to MergedIterator...