[SOLR-12523] Confusing error reporting if backup attempted on non-shared FS

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 7.3.1
• Fix Version/s: 7.5, 8.0
• Component/s: Backup/Restore
• Assignee: Jan Høydahl
• Reporter: Timothy Potter
• Created: 27/Jun/18 19:04
• Updated: 08/Jun/19 15:01
• Resolved: 15/Aug/18 12:09

Description

So I have a large collection with 4 shards across 2 nodes. When I try to back it up with:

curl "http://localhost:8984/solr/admin/collections?action=BACKUP&name=sigs&collection=foo_signals&async=5&location=backups"

I either get:

"5170256188349065":{

    "responseHeader":{

      "status":0,

      "QTime":0},

    "STATUS":"failed",

    "Response":"Failed to backup core=foo_signals_shard1_replica_n2 because org.apache.solr.common.SolrException: Directory to contain snapshots doesn't exist: file:///vol1/cloud84/backups/sigs"},

  "5170256187999044":{

    "responseHeader":{

      "status":0,

      "QTime":0},

    "STATUS":"failed",

    "Response":"Failed to backup core=foo_signals_shard3_replica_n10 because org.apache.solr.common.SolrException: Directory to contain snapshots doesn't exist: file:///vol1/cloud84/backups/sigs"},

or if I create the directory, then I get:

{

  "responseHeader":{

    "status":0,

    "QTime":2},

  "Operation backup caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: The backup directory already exists: file:///vol1/cloud84/backups/sigs/",

  "exception":{

    "msg":"The backup directory already exists: file:///vol1/cloud84/backups/sigs/",

    "rspCode":400},

  "status":{

    "state":"failed",

    "msg":"found [2] in failed tasks"}}

I'm thinking this has to do with having 2 cores from the same collection on the same node but I can't get a collection with 1 shard on each node to work either:

"ec2-52-90-245-38.compute-1.amazonaws.com:8984_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://ec2-52-90-245-38.compute-1.amazonaws.com:8984/solr: Failed to backup core=system_jobs_history_shard2_replica_n6 because org.apache.solr.common.SolrException: Directory to contain snapshots doesn't exist: file:///vol1/cloud84/backups/ugh1"}

What's weird is that replica (system_jobs_history_shard2_replica_n6) is not even on the ec2-52-90-245-38.compute-1.amazonaws.com node! It lives on a different node.