New patch... I think it's close.
This adds "enable/disableAutoIOThrottle" methods to CMS, to have CMS
pick a reasonable IO throttle over time so merges don't fall behind
but also don't suck up all available IO. It's a live setting, and
default is on. CMS.getIORateLimitMBPerSec returns the current
All "merge abort checks" are gone and instead handled by the per-merge
rate limiter that IW sets up for each merge. This gives merge
schedulers "io nice"-like control over each merge thread.
Setting the right IO throttle is a fun control problem (see
http://en.wikipedia.org/wiki/Control_theory), much like the fan in
your laptop that changes its speed depending on internal temperature,
or a factory that must add more workers depending on incoming jobs.
I first tried "open loop" control, trying to set the rate based on
indexing rate or incoming merges rate, but that doesn't work very
well since there are many variables (e.g. CFS on or off) that affect
required MB/sec writing.
So then I switched to a simplistic feedback control: when a merge
arrives, if another merge that's "close" to that same size is still
running, we are falling behind and we aggressively (+20%) increase the
IO throttle. Else, if there is a prior backlog still, leave the rate
unchanged. Else, we decrease it. In my various tests of "tiny
flushed segs" vs "big flushed segs", NRT reopens vs no, CFS or not, 1
2 or 3 merge threads, this approach seems to work well.
I haven't yet tested on spinning disks though ... will have to wait
until I'm back home ... somehow my beast box died while I'm on
vacation! I think fsck must be waiting for me on the console
Forced merges have their own separate throttle (defaults to
I think it's important CMS not have min/max MB/sec throttle control:
I think this just invites disaster when apps set them to inappropriate
values (but I added a protected CMS method "escape hatch" so a
subclass can override the control logic).
I also removed RateLimitedDirectoryWrapper: it's too simplistic and
too dangerous. Finally I cleaned a few things up and improved verbose
infoStream logging so we can see more stats for each merge.