[SOLR-12281] Document the new index back-compat guarantees

• Type: Task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 8.0
• Component/s: documentation
• Assignee: Unassigned
• Reporter: Jan Høydahl
• Created: 26/Apr/18 22:43
• Updated: 25/Oct/21 18:09

Description

Refer to discussions in LUCENE-8264. Since 7.x a lucene segment will carry information about first created version, and in 8.0 Lucene will refuse to read an index created pre-7.0, even if the index is "upgraded" and rewritten in 7.x format.

This new policy breaks with our historic N-1 guarantee, in that you now will be forced to reindex from source every 2 major versions, starting from 8.0. There is a chance that the requirement will be relaxed in 9.0 so that 9.x will be able to read 7.x indices but that I guess depends on what breaking index changes are made before 9.0.

This all must be documented clearly both in CHANGES and in RefGuide. Now we have wordings that you can upgrade 6->7 and then 7->8.

Related to this we should also touch parts of our documentation that deals with stored/docvalues, and add a recommendation to keep things stored because you will need to reindex.

Probably this also affects backup/restore in that if you plan to upgrade to 8.x you will need to make sure that none of your segments are created pre-7.x. And perhaps it affects CDCR as well if one cluster is upgraded before the other etc.

Then finally – and this is beyond just documentation – we should look into better product support for various re-index scenarios. We already have the streaming update() feature, backup/restore, manual /export to json etc. But what about:

• A nice Admin UI on top of the streaming capabilities, where you enter the URL of the remote (old) system and then you get a list of collections that you can select from, and click "GO" and go for lunch, then when you return everything is reindexed into the new cluster.
• Have the install script warn you when doing an in-place upgrade from 7.x to 8.0?
• If Solr 8.x is started and detects a pre-7.0 index, then log big ERROR, and show a big red "UPGRADE" button in the Admin UI. Clicking this button would trigger some magic code that re-builds all collections from source (if stored/DV fields can be found in the old indices).

I've put this as a blocker issue for now, in that we at least need some documentation updates before 8.0, and ideally some tools as well.

Issue Links

• relates to: LUCENE-9669 N-2 compatibility for file formats

Sub-Tasks

• Reference guide section on IndexUpgraderTool needs to be updated: Closed