[LUCENE-7523] UpgradeIndexMergePolicy: beyond one-off use, monster segment avoidance

• Type: Task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Christine Poerschke
• Created: 26/Oct/16 12:22
• Updated: 28/Aug/22 15:05

Description

(Was looking at UpgradeIndexMergePolicy as part of SOLR-9648 and came up with these possibilities here, what do people think?)

Currently one probably would not configure use of the UpgradeIndexMergePolicy (UIMP) permanently since findForcedMerges becomes a no-op after all segments have been upgraded.

• How about adding an optional fallbackToInnerAfterUpgrade flag? That way UIMP.findForcedMerges could fallback to its inner/delegate merge policy's findForcedMerges call after all segments have been upgraded.

Currently UIMP.findForcedMerges identifies all the segments to be upgraded and then asks its inner/delegate merge policy to come up with a MergeSpecification for those segments. If the inner/delegate merge policy does not supply a merge for all the segments to be upgraded then UIMP merges the remaining segments into one new segment. That extra new segment could be quite a large 'monster' segment.

• How about adding an optional upgradeUnmergedSegmentsIndividually flag? That way UIMP.findForcedMerges could upgrade (but not merge) the remaining segments.
• Or indeed should 'upgradeUnmergedSegmentsIndividually' be the default behaviour?

Noticed whilst looking at the code:

• UIMP.findMerges does not pass the mergeTrigger to the inner/delegate merge policy.
  • If we can figure out why that is, let's add a comment to say why that is.
  • Understanding why that is would also be needed before proceeding with beyond one-off use of UIMP.

Issue Links

• relates to: SOLR-9648 Wrap all solr merge policies with SolrMergePolicy
• relates to: SOLR-12259 Robustly upgrade indexes