[LUCENE-6119] Add auto-io-throttle to ConcurrentMergeScheduler

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.0, 6.0
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 17/Dec/14 11:05
• Updated: 28/Aug/22 14:21
• Resolved: 05/Jan/15 15:00

Description

This method returns number of "incoming" bytes IW has written since it
was opened, excluding merging.

It tracks flushed segments, new commits (segments_N), incoming
files/segments by addIndexes, newly written live docs / doc values
updates files.

It's an easy statistic for IW to track and should be useful to help
applications more intelligently set defaults for IO throttling
(RateLimiter).

For example, an application that does hardly any indexing but finally
triggered a large merge can afford to heavily throttle that large
merge so it won't interfere with ongoing searches.

But an application that's causing IW to write new bytes at 50 MB/sec
must set a correspondingly higher IO throttling otherwise merges will
clearly fall behind.

Issue Links

• is related to: LUCENE-10576 ConcurrentMergeScheduler maxThreadCount calculation is artificially low