Major Description
IndexSizeTrigger can be configured to use splitMethod=link (SOLR-12730), which uses hard-linking for creating sub-shards.
However, if the trigger uses aboveBytes condition the resulting sub-shards will not immediately decrease in size, until all of the deleted documents will be expunged (either by gradual merges or by explicit and costly expungeDeletes command). As a result the new sub-shards will still exceed the aboveBytes threshold, which will cause the trigger to keep generating new split requests.
I see two options how to solve this:
- disallow using aboveBytes with splitMethod=link. This unfortunately is a very desirable combination because it monitors the actual index size and uses the fast splitting method.
- calculate an internal estimate of "eventual index size" for an index with deletions, and use this estimate when checking with aboveBytes instead of the real index size. This of course introduces a potentially significant estimation error but allows to properly treat hard-linked sub-shards with deletions as (eventually) significantly smaller than the parent shard.