[SOLR-15673] Incremental backup attempt fails after a shard split operation has completed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.9
• Fix Version/s: 9.0, 8.11, 8.10.2
• Component/s: Backup/Restore
• Assignee: Jason Gerlowski
• Reporter: Sameer
• Created: 04/Oct/21 15:26
• Updated: 17/Nov/21 14:22
• Resolved: 22/Oct/21 14:37

Description

I have been attempting to use the incremental backup API on Solr 8.9.0, but while testing in our product we would occasionally get into a state where all subsequent backup attempts would fail. After some triage we found that it was happening to any collection which had undergone a shard split operation. If we did a backup, completed a shard split operation, then attempted another backup, the second backup would fail with a FileNotFound exception relating to the backup id of the second backup as the error message. 

Steps to reproduce:

 

• Create a new collection with no associated backups
• Run a backup for this collection

• /admin/collections?action=BACKUP&name=myBackupName&collection=myCollectionName&location=/path/to/my/shared/drive

• Run a shard split operation

• /admin/collections?action=SPLITSHARD&collection=name&shard=shardID

• Attempt another backup

 

Expected Outcome:

• If this operation is being blocked intentionally, then I would expect an informative error message explaining why it failed. Otherwise I would expect the backup to complete successfully.

Actual Outcome:

• The backup operation fails with a NoSuchFileException.

NOTE: In the below exception message the number in the file which isn’t found (in this case zk_backup_1) relates to the backup attempt which is currently being attempted. 

java.nio.file.NoSuchFileException: /path/to/my/shared/drive/reproCollectionBackup/reproCollection/zk_backup_1

I tried a few different workaround attempts, but after going through these steps I wasn’t able to run another backup for the collection.

Workaround attempt 1:

• Use the API to delete the backup

• Used the API to purge unused backup files

• Restarted Solr

• Attempted another backup

• Encountered the same failure

Workaround attempt 2:

• Deleted all files in my Solr backup mount location

• Restarted Solr

• Attempted another backup

• Encountered the same failure

Issue Links

• is duplicated by: SOLR-15696 Incremental Backups fail following a splitshard op