Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-10025

SoftDeletesRetentionMergePolicy#numDeletesToMerge caused indexing backlogged

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 8.4
    • Fix Version/s: None
    • Component/s: core/index
    • Lucene Fields:
      New

      Description

      In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.

      if soft deleted docs is very more, and they are also in retention lease,the numDeletesToMerge funcation have  performance issue

      for instance,a update indexing is writing to elasticsearch, then we move one a shard to an other node,If the moving continues for a long time, the size of old shard will become very big,because soft-deleted operations need to held by retention lease. The more soft-deleted documents, the slower the indexing. if the shard size is about 20GB, we can get the below flamegraph

       

       

        Attachments

        1. flamegraph.html
          998 kB
          zhangchao.es
        2. image-2021-07-14-16-52-34-740.png
          824 kB
          zhangchao.es

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zhangchao.es zhangchao.es
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: