Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
8.4
-
None
-
New
Description
In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.
if soft deleted docs is very more, and they are also in retention lease,the numDeletesToMerge funcation have performance issue
for instance,a update indexing is writing to elasticsearch, then we move one a shard to an other node,If the moving continues for a long time, the size of old shard will become very big,because soft-deleted operations need to held by retention lease. The more soft-deleted documents, the slower the indexing. if the shard size is about 20GB, we can get the below flamegraph