[LUCENE-10025] SoftDeletesRetentionMergePolicy#numDeletesToMerge caused indexing backlogged

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 8.4
• Component/s: core/index
• Labels: indexing soft-delete
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: zhangchao.es
• Created: 14/Jul/21 08:32
• Updated: 28/Aug/22 16:23
• Resolved: 26/Jul/21 02:36

Description

In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.

if soft deleted docs is very more, and they are also in retention lease,the numDeletesToMerge funcation have  performance issue

for instance，a update indexing is writing to elasticsearch, then we move one a shard to an other node,If the moving continues for a long time, the size of old shard will become very big,because soft-deleted operations need to held by retention lease. The more soft-deleted documents, the slower the indexing. if the shard size is about 20GB, we can get the below flamegraph