[SOLR-10969] Backup API call failure leaves the backup directory undeleted

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 6.6
• Component/s: Backup/Restore, SolrCloud
• Labels: BACKUP SolrCloud

Environment

Tested on Fedora 24 64-bit (Linux), 8 GB RAM, 2 CPU and
OS: Mac OSX Sierra
Processor: 2.6 GHz Intel Core i5 (64 bit)
RAM: 8 GB

• Assignee: Unassigned
• Reporter: Rohit
• Created: 28/Jun/17 19:26
• Updated: 08/Jun/19 18:20

Description

1. Invoke the Backup API on Solr cloud
2. Backup API fails
3. A directory is created in the location=/path/to/location with name=myBackupname (as specified) in the backup API call.
4. In case of backup API failure the directory should be automatically deleted since, when recalling the API the backup api will fail stating directory already exists

Solr 6.6.0 (fresh install, 4 node solr cluster):
1. Create a collection in Solr called citibike:
http://localhost:8983/solr/admin/collections?action=CREATE&name=citibike&numShards=2&replicationFactor=1&maxShardsPerNode=1&collection.configName=rohit&&createNodeSet=192.168.3.15:7574_solr,192.168.3.15:8983_solr

2. Index 8 documents to Solr collection citibike:
{
"responseHeader":{
"zkConnected":true,
"status":0,
"QTime":10,
"params":{
"q":":",
"indent":"on",
"wt":"json"}},
"response":{"numFound":8,"start":0,"maxScore":1.0,"docs":[

{ "id":"doc1", "_version_":1570643322182041600}

,

{ "id":"doc2", "_version_":1570643322185187328}

,

{ "id":"doc3", "_version_":1570643322185187329}

,

{ "id":"doc5", "_version_":1570643322188333056}

,

{ "id":"doc6", "_version_":1570643322191478784}

,

{ "id":"doc7", "_version_":1570643322191478785}

,

{ "id":"doc8", "_version_":1570643322191478786}

,

{ "id":"doc4", "_version_":1570643322179944448}

]
}}

2. Try to create a backup of the collection with only 8 documents:
http://localhost:8983/solr/admin/collections?action=BACKUP&name=myBackupName&collection=citibike&location=/Users/Rohit/Documents/SolrInstall/backup

{
"responseHeader":

{ "status":500, "QTime":20}

,
"failure":

{ "192.168.3.15:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://192.168.3.15:8983/solr: Failed to backup core=citibike_shard2_replica1 because java.nio.file.NoSuchFileException: /Users/Rohit/Documents/SolrInstall/solr-6.6.0/example/cloud/node1/solr/citibike_shard2_replica1/data/index/segments_8"}

,
"Operation backup caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not backup all replicas",
"exception":

{ "msg":"Could not backup all replicas", "rspCode":500}

,
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"Could not backup all replicas",
"trace":"org.apache.solr.common.SolrException: Could not backup all replicas\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:300)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:237)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:215)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:745)\n",
"code":500}}

Goto path given in location and you will see that a folder with name=myBackupName is created and not cleared off even if the backup API call fails.

Now, re-run the backup API command again. It will fail with exception "The backup directory already exists"

Suggested fix:
1. In case of error delete the directory in the location path automatically since, the partial update is not a correct backup anyways.
2. Add and locationOverwrite parameter. This parameter will clear the old directory and create a backup in the location specified. It will prevent users the need to goto the backup path and to delete the folder automatically.