[SOLR-15696] Incremental Backups fail following a splitshard op

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.9
• Fix Version/s: 9.0, 8.11, 8.10.2
• Component/s: Backup/Restore
• Labels: backups newdev
• Assignee: Jason Gerlowski
• Reporter: Jason Gerlowski
• Created: 15/Oct/21 19:58
• Updated: 17/Nov/21 14:22
• Resolved: 22/Oct/21 14:37

Description

Filing this ticket on behalf of a reporter on the mailing list (Jordan Diehl) who had trouble filing it themselves. See the "Can't create solr jira bugs" thread for more context.

"I have been attempting to use the incremental backup API on Solr 8.9.0, but while testing in our product we would occasionally get into a state where all subsequent backup attempts would fail. After some triage we found that it was happening to any collection which had undergone a shard split operation. If we did a backup, completed a shard split operation, then attempted another backup, the second backup would fail with a FileNotFound exception relating to the backup id of the second backup as the error message."

Steps to Reproduce:

Run the script found here in a clean 8.9.0 download. In essence, it (1) creates and fills a collection (2) performs a backup (successfully), (3) splits a shard in the collection, and (4) triggers another backup (which fails).

Expected Behavior

"If this operation is being blocked intentionally, then I would expect an informative error message explaining why it failed. Otherwise I would expect the backup to complete successfully."

Actual Behavior

"The backup operation fails with a NoSuchFileException."

{

  "responseHeader":{
    "status":500,
    "QTime":54},
  "failure":{
"MYIPADDRESS:31018_solr":"org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:Error from server at null: Error handling 'BACKUPCORE' action"},
  "Operation backup caused exception:":"java.nio.file.NoSuchFileException:java.nio.file.NoSuchFileException: /opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1",
  "exception":{
    "msg":"/opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1",
    "rspCode":-1},
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"/opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1",
    "trace":"org.apache.solr.common.SolrException: /opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1\n\tat org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:65)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:301)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:257)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:836)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:800)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:545)...<snip>...",
    "code":500}}

Issue Links

• duplicates: SOLR-15673 Incremental backup attempt fails after a shard split operation has completed
• links to: GitHub Pull Request #351