Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8162

Make it possible to throttle (Tiered)MergePolicy when commit rate is high

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: trunk
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      As discussed in a recent mailing list thread [1] and observed in a project using Lucene (see OAK-5192 and OAK-6710), it is sometimes helpful to throttle the aggressiveness of (Tiered)MergePolicy when commit rate is high.

      In the case of Apache Jackrabbit Oak a dedicated MergePolicy was implemented [2].

      That MP doesn't merge in case the number of segments is below a certain threshold (e.g. 30) and commit rate (docs per sec and MB per sec) is high (e.g. above 1000 doc / sec , 5MB / sec).

      In such impl, the commit rate thresholds adapt to average commit rate by means of single exponential smoothing.

      The results in that specific case looked encouraging as it brought a 5% perf improvement in querying and ~10% reduced IO. However Oak has some specifics which might not fit in other scenarios. Anyway it could be interesting to see how this behaves in plain Lucene scenario.

      [1]http://markmail.org/message/re3ifmq2664bqfjk

      [2]https://github.com/apache/jackrabbit-oak/blob/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/writer/CommitMitigatingTieredMergePolicy.java

        Attachments

        1. LUCENE-8162.0.patch
          1 kB
          Tommaso Teofili

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              teofili Tommaso Teofili
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: