[SOLR-13216] Trouble restoring a collection

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 6.6.5
• Component/s: Backup/Restore
• Assignee: Unassigned
• Reporter: Roy Perkins
• Created: 04/Feb/19 18:06
• Updated: 08/Jun/19 14:58

Description

I'm having a weird issue when attempting to restore a collection from our prod cluster to our staging cluster.  The restore seems to be moving along normally, and then right at the end, the data gets dumped altogether.

Below is the command I use to restore:

curl -s "http://localhost:8983/solr/admin/collections?action=RESTORE&name=slprod-02-04-2019&location=/mnt/solr_backups/slprod&collection=slprod-02-04-2019&maxShardsPerNode=1&replicationFactor=1&async=1000"

Below are relevant messages in the logs: 

2019-02-04 12:51:57.465 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is2_Lucene50_0.tip to restore directory
2019-02-04 12:51:57.524 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is3.fdt to restore directory
2019-02-04 12:51:57.590 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is3.fdx to restore directory
2019-02-04 12:51:57.642 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is3.fnm to restore directory
2019-02-04 12:51:57.707 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is3.si to restore directory
2019-02-04 12:51:57.760 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is3_Lucene50_0.doc to restore directory
2019-02-04 12:51:57.812 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is3_Lucene50_0.tim to restore directory
2019-02-04 12:51:57.878 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is3_Lucene50_0.tip to restore directory
2019-02-04 12:51:57.936 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4.fdt to restore directory
2019-02-04 12:51:58.003 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4.fdx to restore directory
2019-02-04 12:51:58.057 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4.fnm to restore directory
2019-02-04 12:51:58.124 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4.nvd to restore directory
2019-02-04 12:51:58.191 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4.nvm to restore directory
2019-02-04 12:51:58.244 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4.si to restore directory
2019-02-04 12:51:58.298 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4_Lucene50_0.doc to restore directory
2019-02-04 12:51:58.350 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4_Lucene50_0.pos to restore directory
2019-02-04 12:51:58.402 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4_Lucene50_0.tim to restore directory
2019-02-04 12:51:58.467 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file _406is4_Lucene50_0.tip to restore directory
2019-02-04 12:51:58.520 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Copying file segments_3x02 to restore directory
2019-02-04 12:51:58.573 INFO (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.c.SolrCore Updating index properties... index=restore.20190204124745615
2019-02-04 12:51:58.599 WARN (parallelCoreAdminExecutor-5-thread-2-processing-n:solrmcstg11.domain:8983_solr 100019616170409755642 RESTORECORE) [ ] o.a.s.h.RestoreCore Could not switch to restored index. Rolling back to the current index
org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data/solr/slprod-02-04-2019_shard3_replica0/data/restore.20190204124745615/segments_3x02")))
{{ at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:290)}}
{{ at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:930)}}
{{ at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:118)}}
{{ at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:93)}}
{{ at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)}}
{{ at org.apache.solr.update.DefaultSolrCoreState.changeWriter(DefaultSolrCoreState.java:220)}}
{{ at org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:229)}}
{{ at org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:726)}}
{{ at org.apache.solr.handler.RestoreCore.doRestore(RestoreCore.java:108)}}
{{ at org.apache.solr.handler.admin.RestoreCoreOp.execute(RestoreCoreOp.java:65)}}
{{ at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)}}
{{ at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)}}
{{ at org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:182)}}
{{ at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)}}
{{ at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)}}
{{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}}
{{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}}
{{ at java.lang.Thread.run(Thread.java:745)}}
Caused by: java.nio.file.NoSuchFileException: /data/solr/slprod-02-04-2019_shard3_replica0/data/restore.20190204124745615/_406hsh.si
{{ at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)}}
{{ at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)}}
{{ at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)}}
{{ at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)}}
{{ at java.nio.channels.FileChannel.open(FileChannel.java:287)}}
{{ at java.nio.channels.FileChannel.open(FileChannel.java:335)}}
{{ at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)}}
{{ at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:192)}}
{{ at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)}}

{{ at org.apache.lucene.codecs.lucene62.Lucene62SegmentInfoFormat.read(Lucene62SegmentInfoFormat.java:89)}}
{{ at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:357)}}
{

{ at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:288)}

}