This issue illustrates why the ram dir approach can be useful,
because small segment merges compete with large segment merges
for IO, which can spike the turnaround time. With a ram dir, the
small segments are held in RAM until they're large enough to be
placed onto disk. They can then be given the same IO priority as
the other merging segments which should result in consistent
reopen times.Make CMS smarter about thread priorities
I'm still not convinced we should game the OS, here.
Ie, the small segments are likely still in the IO cache, so
we're probably not really competing for IO on the smaller merges.
I think it's when an immense merge is running that we're in trouble.
EG I see very long NRT reopen times when such a merge is running.
I've been wondering whether we need to take IO prioritization into our
own hands. EG, instead of relying [only] on thread priorities for
CMS, somehow have the big merge pause until the small merge can
complete. This would really be the best way to implement the "allow
at most 1 merge to run at once". I guess we may be able to override
the mergeAbort to implement this...
I think a similar "emulate IO prioritization" may help when, eg, a
large merge/optimize is interfering with ongoing searches. Yes, some
of the cost is because the merge is evicting IO cache, but for large
indexes that don't fit in RAM, there must be some cost to sharing the
disk heads, too...