Solr
  1. Solr
  2. SOLR-6637

Solr should have a way to restore a core

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      We have a core backup command which backs up the index. We should have a restore command too.

      This would restore any named snapshots created by the replication handlers backup command.

      While working on this patch right now I realized that during backup we only backup the index. Should we backup the conf files also? Any thoughts? I could separate Jira for this.

      1. SOLR-6637.patch
        51 kB
        Varun Thacker
      2. SOLR-6637.patch
        51 kB
        Noble Paul
      3. SOLR-6637.patch
        52 kB
        Varun Thacker
      4. SOLR-6637.patch
        50 kB
        Varun Thacker
      5. SOLR-6637.patch
        46 kB
        Varun Thacker
      6. SOLR-6637.patch
        24 kB
        Varun Thacker
      7. SOLR-6637.patch
        18 kB
        Varun Thacker
      8. SOLR-6637.patch
        18 kB
        Varun Thacker
      9. SOLR-6637.patch
        16 kB
        Varun Thacker
      10. SOLR-6637.patch
        9 kB
        Varun Thacker

        Issue Links

          Activity

          Hide
          Varun Thacker added a comment -

          Patch which adds functionality to restore a core. I am starting to work on a test case.

          In parallel I am also working to provide a patch for SOLR-5750.

          Show
          Varun Thacker added a comment - Patch which adds functionality to restore a core. I am starting to work on a test case. In parallel I am also working to provide a patch for SOLR-5750 .
          Hide
          Varun Thacker added a comment -

          Patch with a working test case.

          We need to discuss the scenario where we hit an exception while restoring. i.e We copy all segment files from the backup location to the index location overwriting any segment files with the same name. Now while copying if we hit an exception then the index can get corrupt.

          Is there any way to deal with it or do we just need to document this scenario.

          Show
          Varun Thacker added a comment - Patch with a working test case. We need to discuss the scenario where we hit an exception while restoring. i.e We copy all segment files from the backup location to the index location overwriting any segment files with the same name. Now while copying if we hit an exception then the index can get corrupt. Is there any way to deal with it or do we just need to document this scenario.
          Hide
          Noble Paul added a comment -

          can you just give a quick writeup on the steps done by this operation

          Show
          Noble Paul added a comment - can you just give a quick writeup on the steps done by this operation
          Hide
          Varun Thacker added a comment -

          Sure.

          1. We can issue only one restore command for a core.
          2. If the "location" is not provided in the query string then we default it to core.getDataDir() . We do the same while taking backups hence this behaviour instead of making it mandatory.

          In RestoreCore we do the following -
          1. Close the current index writer
          2. Copy over all files from the backup location to the current index directory.
          3. Remove any files in the current directory which does not belong to the segment .( Hmm.. this might not work when there are multiple segments ) Maybe instead of using

          SegmentInfos.readCommit(dir, segmentFileName)
          

          we just need to do something simple i.e just keep track of what files were before and what files are currently present and remove the unwanted. I will check if this needs to be fixed.

          4. Open a new writer and searcher against the index before we return restore success.

          We track the status of the restore with a "restorestatus" API. This checks if the future is finished executing or not.

          Show
          Varun Thacker added a comment - Sure. 1. We can issue only one restore command for a core. 2. If the "location" is not provided in the query string then we default it to core.getDataDir() . We do the same while taking backups hence this behaviour instead of making it mandatory. In RestoreCore we do the following - 1. Close the current index writer 2. Copy over all files from the backup location to the current index directory. 3. Remove any files in the current directory which does not belong to the segment .( Hmm.. this might not work when there are multiple segments ) Maybe instead of using SegmentInfos.readCommit(dir, segmentFileName) we just need to do something simple i.e just keep track of what files were before and what files are currently present and remove the unwanted. I will check if this needs to be fixed. 4. Open a new writer and searcher against the index before we return restore success. We track the status of the restore with a "restorestatus" API. This checks if the future is finished executing or not.
          Hide
          Varun Thacker added a comment -

          Updated patch. Improved the test case and fixed the bugs that it uncovered.

          Still fighting one issue - I keep getting this error when the index directory tries to close all the resources. Trying to figure out what is the underlying problem.

          MockDirectoryWrapper: cannot close: there are still open files: {_0.cfs=1, _1.cfs=1}
          
          Show
          Varun Thacker added a comment - Updated patch. Improved the test case and fixed the bugs that it uncovered. Still fighting one issue - I keep getting this error when the index directory tries to close all the resources. Trying to figure out what is the underlying problem. MockDirectoryWrapper: cannot close: there are still open files: {_0.cfs=1, _1.cfs=1}
          Hide
          Varun Thacker added a comment -

          MockDirectoryWrapper: cannot close: there are still open files: {_0.cfs=1, _1.cfs=1}

          Patch which fixes this. Looks like we can't use the try with resource block to get indexDir from the directoryFactory as we need to call release() instead of closing it.

          Show
          Varun Thacker added a comment - MockDirectoryWrapper: cannot close: there are still open files: {_0.cfs=1, _1.cfs=1} Patch which fixes this. Looks like we can't use the try with resource block to get indexDir from the directoryFactory as we need to call release() instead of closing it.
          Hide
          David Smiley added a comment -

          FYI I already created an issue in JIRA for this: https://issues.apache.org/jira/browse/SOLR-4545

          Show
          David Smiley added a comment - FYI I already created an issue in JIRA for this: https://issues.apache.org/jira/browse/SOLR-4545
          Hide
          Varun Thacker added a comment -

          David Smiley I had not seen that issue previously. Should we move the work there ?

          A proposed "restore" command to the replication handler should allow specifying a directory, or an "as-of" date; otherwise you'd get the most recent snapshot.

          My approach here has been to allow restoring named snapshots ( SOLR-5340 ) only. We can add functionality that says that if the "name" is not provided then we restore the most recent snapshot.

          Show
          Varun Thacker added a comment - David Smiley I had not seen that issue previously. Should we move the work there ? A proposed "restore" command to the replication handler should allow specifying a directory, or an "as-of" date; otherwise you'd get the most recent snapshot. My approach here has been to allow restoring named snapshots ( SOLR-5340 ) only. We can add functionality that says that if the "name" is not provided then we restore the most recent snapshot.
          Hide
          Noble Paul added a comment -

          Varun Thacker Can I not restore the data to another core?

          If the "location" is not provided in the query string then we default it to core.getDataDir()

          what is the usecase for restoring from the dataDir itself?

          Remove any files in the current directory which does not belong to the segment .

          DON'T DO THIS

          There is a mechanism using for loading the index from another directory in the same core.

          Show
          Noble Paul added a comment - Varun Thacker Can I not restore the data to another core? If the "location" is not provided in the query string then we default it to core.getDataDir() what is the usecase for restoring from the dataDir itself? Remove any files in the current directory which does not belong to the segment . DON'T DO THIS There is a mechanism using for loading the index from another directory in the same core.
          Hide
          Varun Thacker added a comment -

          what is the usecase for restoring from the dataDir itself?

          What I meant here was - if location param is not provided it would see if the backup index is present under dataDir/backupName .

          Remove any files in the current directory which does not belong to the segment .

          This is what I do here - Once all the files from the backup location have been successfully copied over the current index, there might be extra segment files from the current index lying around. It gets cleaned up in cleanupOldIndexFiles() where we take the name of the segment file from the backup index and see which files are extra. We then remove these extra segment files.

          Show
          Varun Thacker added a comment - what is the usecase for restoring from the dataDir itself? What I meant here was - if location param is not provided it would see if the backup index is present under dataDir/backupName . Remove any files in the current directory which does not belong to the segment . This is what I do here - Once all the files from the backup location have been successfully copied over the current index, there might be extra segment files from the current index lying around. It gets cleaned up in cleanupOldIndexFiles() where we take the name of the segment file from the backup index and see which files are extra. We then remove these extra segment files.
          Hide
          David Smiley added a comment -

          David Smiley I had not seen that issue previously. Should we move the work there ?

          No, it's too late now. Next time please search for an existing issue. SOLR-4545 can be closed as a duplicate so long as you can restore a snapshot without being required to specify its name. A timestamp would be nice.

          Show
          David Smiley added a comment - David Smiley I had not seen that issue previously. Should we move the work there ? No, it's too late now. Next time please search for an existing issue. SOLR-4545 can be closed as a duplicate so long as you can restore a snapshot without being required to specify its name. A timestamp would be nice.
          Hide
          Varun Thacker added a comment -

          New patch.

          SOLR-4545 can be closed as a duplicate so long as you can restore a snapshot without being required to specify its name. A timestamp would be nice.

          Done. How it works is if the backup name is provided it restores the named snapshot. Otherwise it looks in the backup directory for the oldest timestamp backup and restores that.

          Additionally the patch improves the test and shuts down the executor service correctly.

          Show
          Varun Thacker added a comment - New patch. SOLR-4545 can be closed as a duplicate so long as you can restore a snapshot without being required to specify its name. A timestamp would be nice. Done. How it works is if the backup name is provided it restores the named snapshot. Otherwise it looks in the backup directory for the oldest timestamp backup and restores that. Additionally the patch improves the test and shuts down the executor service correctly.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks for the patch Varun.

          A few comments on the patch and the feature in general:

          1. The usage of restoreLock is wrong. If you issue a second restore then you'll get a IllegalMonitorStateException. The thread which acquires the lock must be the one which releases it. So just calling unlock from another request thread won't work. The RestoreCore thread must be the one to acquire and release the lock (in a finally clause)
          2. You should cancel the future interruptibly in the close hook. Just executor service's awaitTermination is not enough.
          3. There are no guarantees here that the previous backup was actually complete before you start restoring it. This might need another issue to fix the snapshoot command itself.
          4. If restoreFuture is null (no restore has ever been requested), the restore status will return "in progress".
          5. Considering that a restore is a full replace of the older index, we can just use the same index.properties method that SnapPuller uses to ask SolrCore to reload with a new index directory. That would eliminate the copying around files to the index directory.
          6. There should be an option to not copy files from the source location at all and instead to just use it directly as the new index directory.
          7. We should be able to rollback to original state (old directory) if the new index fails integrity checks.

          We need tests for all these scenarios. I'd also like to see more code being refactored and shared between Snapshoot, Snappuller and RestoreCore.

          Show
          Shalin Shekhar Mangar added a comment - Thanks for the patch Varun. A few comments on the patch and the feature in general: The usage of restoreLock is wrong. If you issue a second restore then you'll get a IllegalMonitorStateException. The thread which acquires the lock must be the one which releases it. So just calling unlock from another request thread won't work. The RestoreCore thread must be the one to acquire and release the lock (in a finally clause) You should cancel the future interruptibly in the close hook. Just executor service's awaitTermination is not enough. There are no guarantees here that the previous backup was actually complete before you start restoring it. This might need another issue to fix the snapshoot command itself. If restoreFuture is null (no restore has ever been requested), the restore status will return "in progress". Considering that a restore is a full replace of the older index, we can just use the same index.properties method that SnapPuller uses to ask SolrCore to reload with a new index directory. That would eliminate the copying around files to the index directory. There should be an option to not copy files from the source location at all and instead to just use it directly as the new index directory. We should be able to rollback to original state (old directory) if the new index fails integrity checks. We need tests for all these scenarios. I'd also like to see more code being refactored and shared between Snapshoot, Snappuller and RestoreCore.
          Hide
          Noble Paul added a comment -

          I believe having a new class RestoreCore for this small functionality is overkill. It can be made part of SnapPuller. If you refactor a bit , both snappull and restore will have a lot of commonalities

          Show
          Noble Paul added a comment - I believe having a new class RestoreCore for this small functionality is overkill. It can be made part of SnapPuller. If you refactor a bit , both snappull and restore will have a lot of commonalities
          Hide
          Greg Solovyev added a comment -

          I have been looking for this functionality, but with a slight twist where index files for the core need to be shipped over the network rather than provided on storage local to the Solr instance where the core is being restored. I have an implementation of CoreRestoreHandler that takes index files over HTTP as ContentStreams. Should I submit it to this ticket as a patch?

          Show
          Greg Solovyev added a comment - I have been looking for this functionality, but with a slight twist where index files for the core need to be shipped over the network rather than provided on storage local to the Solr instance where the core is being restored. I have an implementation of CoreRestoreHandler that takes index files over HTTP as ContentStreams. Should I submit it to this ticket as a patch?
          Hide
          Noble Paul added a comment -

          Please post the patch anyway

          Show
          Noble Paul added a comment - Please post the patch anyway
          Hide
          Varun Thacker added a comment -

          Thanks Shalin for the review! I'm sorry it took this long for me to resume working on this.

          Updated patch.

          The usage of restoreLock is wrong. If you issue a second restore then you'll get a IllegalMonitorStateException. The thread which acquires the lock must be the one which releases it.So just calling unlock from another request thread won't work. The RestoreCore thread must be the one to acquire and release the lock (in a finally clause)

          The restore lock is acquired and unlocked by the RestoreCore thread.

          You should cancel the future interruptibly in the close hook. Just executor service's awaitTermination is not enough.

          Changed it to ExecutorUtil.shutdownNowAndAwaitTermination(restoreExecutor);

          If restoreFuture is null (no restore has ever been requested), the restore status will return "in progress".

          Fixed.

          Considering that a restore is a full replace of the older index, we can just use the same index.properties method that SnapPuller uses to ask SolrCore to reload with a new index directory. That would eliminate the copying around files to the index directory.

          Files a copied over to a new restore directory and the using the index.properties method we switch the writer/searcher to this directory. If the same backed up segment files is present in the current index directory then that is used as the source for copying.

          We should be able to rollback to original state (old directory) if the new index fails integrity checks.

          It rolls back to the original index if we are uable to open the restored index.

          Spent a few hours further refactoring SnapPuller so that RestoreCore could use the index diff copying logic in there. But then I realized that SnapPuller always assumes that the source it's fetching from is always ahead of it's current state ( so we don't need to do cleanups if copying into the same directory fails). This would always not be the case when Restoring a core so decided not to go down that path.

          Tests passed and precommit passes after this svn propset svn:eol-style native ./solr/core/src/java/org/apache/solr/handler/OldBackupDirectory.java ./solr/core/src/java/org/apache/solr/handler/ReplicationUtils.java ./solr/core/src/java/org/apache/solr/handler/RestoreCore.java

          We'll need a slightly different patch for 5x which takes back-compat into consideration while restoring of files in RestoreCore.doRestore()

          Show
          Varun Thacker added a comment - Thanks Shalin for the review! I'm sorry it took this long for me to resume working on this. Updated patch. The usage of restoreLock is wrong. If you issue a second restore then you'll get a IllegalMonitorStateException. The thread which acquires the lock must be the one which releases it.So just calling unlock from another request thread won't work. The RestoreCore thread must be the one to acquire and release the lock (in a finally clause) The restore lock is acquired and unlocked by the RestoreCore thread. You should cancel the future interruptibly in the close hook. Just executor service's awaitTermination is not enough. Changed it to ExecutorUtil.shutdownNowAndAwaitTermination(restoreExecutor); If restoreFuture is null (no restore has ever been requested), the restore status will return "in progress". Fixed. Considering that a restore is a full replace of the older index, we can just use the same index.properties method that SnapPuller uses to ask SolrCore to reload with a new index directory. That would eliminate the copying around files to the index directory. Files a copied over to a new restore directory and the using the index.properties method we switch the writer/searcher to this directory. If the same backed up segment files is present in the current index directory then that is used as the source for copying. We should be able to rollback to original state (old directory) if the new index fails integrity checks. It rolls back to the original index if we are uable to open the restored index. Spent a few hours further refactoring SnapPuller so that RestoreCore could use the index diff copying logic in there. But then I realized that SnapPuller always assumes that the source it's fetching from is always ahead of it's current state ( so we don't need to do cleanups if copying into the same directory fails). This would always not be the case when Restoring a core so decided not to go down that path. Tests passed and precommit passes after this svn propset svn:eol-style native ./solr/core/src/java/org/apache/solr/handler/OldBackupDirectory.java ./solr/core/src/java/org/apache/solr/handler/ReplicationUtils.java ./solr/core/src/java/org/apache/solr/handler/RestoreCore.java We'll need a slightly different patch for 5x which takes back-compat into consideration while restoring of files in RestoreCore.doRestore()
          Hide
          Varun Thacker added a comment -

          Updated patch.

          • Moved all the tests into a separate class TestRestoreCore
          • Uses lucene checksums to to verify if a segment file is same when preferring the local copy.
          Show
          Varun Thacker added a comment - Updated patch. Moved all the tests into a separate class TestRestoreCore Uses lucene checksums to to verify if a segment file is same when preferring the local copy.
          Hide
          Varun Thacker added a comment -

          Updated the patch to trunk

          Show
          Varun Thacker added a comment - Updated the patch to trunk
          Hide
          Noble Paul added a comment -

          I have removed the reentrant lock and the synchronized is good enough . Please take a look

          Show
          Noble Paul added a comment - I have removed the reentrant lock and the synchronized is good enough . Please take a look
          Hide
          Varun Thacker added a comment -

          Patch which folds in Noble's changes. I think its ready.

          If there are no objections I will commit this to trunk tomorrow and wait a few days before backporting to branch_5x.

          Also the patch would be slightly different on branch_5x as lucene checksums are not guaranteed to exist since there can be pre 4.8 segments

          Show
          Varun Thacker added a comment - Patch which folds in Noble's changes. I think its ready. If there are no objections I will commit this to trunk tomorrow and wait a few days before backporting to branch_5x. Also the patch would be slightly different on branch_5x as lucene checksums are not guaranteed to exist since there can be pre 4.8 segments
          Hide
          ASF subversion and git services added a comment -

          Commit 1671022 from Varun Thacker in branch 'dev/trunk'
          [ https://svn.apache.org/r1671022 ]

          SOLR-6637: Solr should have a way to restore a core

          Show
          ASF subversion and git services added a comment - Commit 1671022 from Varun Thacker in branch 'dev/trunk' [ https://svn.apache.org/r1671022 ] SOLR-6637 : Solr should have a way to restore a core
          Hide
          ASF subversion and git services added a comment -

          Commit 1671400 from Varun Thacker in branch 'dev/trunk'
          [ https://svn.apache.org/r1671400 ]

          SOLR-6637: Better error handling when retrieving checksums (Fixes Policeman Jenkins Failure #2134)

          Show
          ASF subversion and git services added a comment - Commit 1671400 from Varun Thacker in branch 'dev/trunk' [ https://svn.apache.org/r1671400 ] SOLR-6637 : Better error handling when retrieving checksums (Fixes Policeman Jenkins Failure #2134)
          Hide
          ASF subversion and git services added a comment -

          Commit 1671919 from Varun Thacker in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1671919 ]

          SOLR-6637: Solr should have a way to restore a core (merged from trunk r1671022 r1671400)

          Show
          ASF subversion and git services added a comment - Commit 1671919 from Varun Thacker in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1671919 ] SOLR-6637 : Solr should have a way to restore a core (merged from trunk r1671022 r1671400)
          Hide
          Varun Thacker added a comment -

          Thanks everybody!

          Show
          Varun Thacker added a comment - Thanks everybody!
          Hide
          Mark Miller added a comment -

          Looks like this issue caused SOLR-7364.

          You can't assume a local filesystem when dealing with paths.

          Show
          Mark Miller added a comment - Looks like this issue caused SOLR-7364 . You can't assume a local filesystem when dealing with paths.
          Hide
          Varun Thacker added a comment -

          Oh that looks like my mistake. Sorry about that.

          We need to change this tmpIndex = Paths.get(solrCore.getDataDir(), tmpIdxDirName).toString(); to Paths.get(solrCore.getDataDir()).resolve(tmpIdxDirName).toString();

          There were two more places which need to be corrected. I'll fix them.

          Show
          Varun Thacker added a comment - Oh that looks like my mistake. Sorry about that. We need to change this tmpIndex = Paths.get(solrCore.getDataDir(), tmpIdxDirName).toString(); to Paths.get(solrCore.getDataDir()).resolve(tmpIdxDirName).toString(); There were two more places which need to be corrected. I'll fix them.
          Hide
          Mark Miller added a comment -

          What's the motivation for using Path at all? This doesn't seem right as it's a local filesystem thing? How do we know that is going to work with any DirectoryFactory impl?

          Show
          Mark Miller added a comment - What's the motivation for using Path at all? This doesn't seem right as it's a local filesystem thing? How do we know that is going to work with any DirectoryFactory impl?
          Hide
          Varun Thacker added a comment -

          How do we know that is going to work with any DirectoryFactory impl?

          You're right. It should stay the way it was originally - tmpIndex = solrCore.getDataDir() + tmpIdxDirName; - The HDFS tests rightly fail otherwise.

          Also this got me thinking that backups or restores won't work when using HDFSDir since we write out the backup index using SimpleFSDirectory. I will test it out and create another Jira for it.

          Should we add HDFSDir to the list of directories in LuceneTestCase?

          Show
          Varun Thacker added a comment - How do we know that is going to work with any DirectoryFactory impl? You're right. It should stay the way it was originally - tmpIndex = solrCore.getDataDir() + tmpIdxDirName; - The HDFS tests rightly fail otherwise. Also this got me thinking that backups or restores won't work when using HDFSDir since we write out the backup index using SimpleFSDirectory. I will test it out and create another Jira for it. Should we add HDFSDir to the list of directories in LuceneTestCase?
          Hide
          ASF subversion and git services added a comment -

          Commit 1672620 from Varun Thacker in branch 'dev/trunk'
          [ https://svn.apache.org/r1672620 ]

          SOLR-6637: can't assume a local filesystem when dealing with paths

          Show
          ASF subversion and git services added a comment - Commit 1672620 from Varun Thacker in branch 'dev/trunk' [ https://svn.apache.org/r1672620 ] SOLR-6637 : can't assume a local filesystem when dealing with paths
          Hide
          ASF subversion and git services added a comment -

          Commit 1672641 from Varun Thacker in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1672641 ]

          SOLR-6637: can't assume a local filesystem when dealing with paths (merged from trunk r1672620)

          Show
          ASF subversion and git services added a comment - Commit 1672641 from Varun Thacker in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1672641 ] SOLR-6637 : can't assume a local filesystem when dealing with paths (merged from trunk r1672620)
          Hide
          Varun Thacker added a comment -

          Resolving this issue. Fixed the broken Path calls. We can tackle backup/restore working correctly on HDFS here SOLR-7374

          Show
          Varun Thacker added a comment - Resolving this issue. Fixed the broken Path calls. We can tackle backup/restore working correctly on HDFS here SOLR-7374
          Hide
          Varun Thacker added a comment -

          Need to fix the CHANGES entry.

          Show
          Varun Thacker added a comment - Need to fix the CHANGES entry.
          Hide
          ASF subversion and git services added a comment -

          Commit 1673420 from Varun Thacker in branch 'dev/trunk'
          [ https://svn.apache.org/r1673420 ]

          SOLR-6637: improve CHANGES entry + fix wrong usage of path in snapshooter

          Show
          ASF subversion and git services added a comment - Commit 1673420 from Varun Thacker in branch 'dev/trunk' [ https://svn.apache.org/r1673420 ] SOLR-6637 : improve CHANGES entry + fix wrong usage of path in snapshooter
          Hide
          ASF subversion and git services added a comment -

          Commit 1673423 from Varun Thacker in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1673423 ]

          SOLR-6637: improve CHANGES entry + fix wrong usage of path in snapshooter (merged from trunk r1673420)

          Show
          ASF subversion and git services added a comment - Commit 1673423 from Varun Thacker in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1673423 ] SOLR-6637 : improve CHANGES entry + fix wrong usage of path in snapshooter (merged from trunk r1673420)
          Hide
          Varun Thacker added a comment -

          Sample Restore API Usage -
          http://localhost:8983/solr/techproducts/replication?command=restore&name=backup_name

          Parameters for the api -
          location - The location where the backup index. If not specified it looks for backups in Solr's data directory.
          name - The name of the backed up index snapshot to be restored. If the name is not provided it looks for backups with snapshot.<timestamp> format in the location directory. It picks the latest timestamp backup in that case.

          You can check the status of the operation with the following call -
          http://localhost:8983/solr/techproducts/replication?command=restorestatus

          Sample output for the restore API -

          <?xml version="1.0" encoding="UTF-8"?>
          <response>
             <lst name="responseHeader">
                <int name="status">0</int>
                <int name="QTime">0</int>
             </lst>
             <lst name="restorestatus">
                <str name="snapshotName">snapshot.ab</str>
                <str name="status">success</str>
             </lst>
          </response>
          

          The status value can be "In Progress" , "success" or "failed". If it failed then an "exception" will also be sent in the response.

          Show
          Varun Thacker added a comment - Sample Restore API Usage - http://localhost:8983/solr/techproducts/replication?command=restore&name=backup_name Parameters for the api - location - The location where the backup index. If not specified it looks for backups in Solr's data directory. name - The name of the backed up index snapshot to be restored. If the name is not provided it looks for backups with snapshot.<timestamp> format in the location directory. It picks the latest timestamp backup in that case. You can check the status of the operation with the following call - http://localhost:8983/solr/techproducts/replication?command=restorestatus Sample output for the restore API - <?xml version= "1.0" encoding= "UTF-8" ?> <response> <lst name= "responseHeader" > < int name= "status" >0</ int > < int name= "QTime" >0</ int > </lst> <lst name= "restorestatus" > <str name= "snapshotName" >snapshot.ab</str> <str name= "status" >success</str> </lst> </response> The status value can be "In Progress" , "success" or "failed". If it failed then an "exception" will also be sent in the response.
          Hide
          Varun Thacker added a comment -

          Improved the CHANGES entry, added API examples. The Ref Guide will also be updated after the 5.1 Ref Guide release.

          Show
          Varun Thacker added a comment - Improved the CHANGES entry, added API examples. The Ref Guide will also be updated after the 5.1 Ref Guide release.
          Hide
          Greg Solovyev added a comment -

          Would it make sense to add an ability to upload index files to target Solr instance via HTTP using ContentStreams or is it a bad idea for any reason?

          Show
          Greg Solovyev added a comment - Would it make sense to add an ability to upload index files to target Solr instance via HTTP using ContentStreams or is it a bad idea for any reason?
          Hide
          Greg Solovyev added a comment -

          Posted my patch to this subtask: https://issues.apache.org/jira/browse/SOLR-7567

          Show
          Greg Solovyev added a comment - Posted my patch to this subtask: https://issues.apache.org/jira/browse/SOLR-7567
          Hide
          Anshum Gupta added a comment -

          Bulk close for 5.2.0.

          Show
          Anshum Gupta added a comment - Bulk close for 5.2.0.
          Hide
          David Smiley added a comment -

          While working on SOLR-5750 (Collection level backup/restore), I read the code for RestoreCore and I'm confused why the "prefer local copy" logic is expressly avoided for these file extensions: .si, .liv, and the segments_. Why these files? It seems arbitrary and there are no comments saying why. Do you know Varun Thacker?

          Show
          David Smiley added a comment - While working on SOLR-5750 (Collection level backup/restore), I read the code for RestoreCore and I'm confused why the "prefer local copy" logic is expressly avoided for these file extensions: .si, .liv, and the segments_. Why these files? It seems arbitrary and there are no comments saying why. Do you know Varun Thacker ?
          Hide
          Varun Thacker added a comment -

          Hi David,

          The idea came when we were working on issues are replication causing index corruption : Here is a related Jira comment : https://issues.apache.org/jira/browse/SOLR-6920?focusedCommentId=14309370&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14309370

          The actual impl. is in {[IndexFetcher#filesToAlwaysDownloadIfNoChecksums}} . Maybe we could reuse that method ?

          Show
          Varun Thacker added a comment - Hi David, The idea came when we were working on issues are replication causing index corruption : Here is a related Jira comment : https://issues.apache.org/jira/browse/SOLR-6920?focusedCommentId=14309370&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14309370 The actual impl. is in {[IndexFetcher#filesToAlwaysDownloadIfNoChecksums}} . Maybe we could reuse that method ?
          Hide
          David Smiley added a comment -

          Maybe we could reuse that method?

          +1 Most definitely. The change to re-use this method is minor enough that I'll just slide it into SOLR-5750.

          Show
          David Smiley added a comment - Maybe we could reuse that method? +1 Most definitely. The change to re-use this method is minor enough that I'll just slide it into SOLR-5750 .
          Hide
          Hrishikesh Gadre added a comment -

          Varun Thacker Shalin Shekhar Mangar

          Please take a look at this code snippet,

          https://github.com/apache/lucene-solr/blob/4193e60b9fc1ff12df2267778213ae3b0f04fb84/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L605-L617

          Is there a specific reason why we have used "SegmentInfos.readCommit(...)" method instead of just using "commit.getFileNames()" ? It seems equivalent as per my code understanding but not sure if I have missed anything...

          Show
          Hrishikesh Gadre added a comment - Varun Thacker Shalin Shekhar Mangar Please take a look at this code snippet, https://github.com/apache/lucene-solr/blob/4193e60b9fc1ff12df2267778213ae3b0f04fb84/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L605-L617 Is there a specific reason why we have used "SegmentInfos.readCommit(...)" method instead of just using "commit.getFileNames()" ? It seems equivalent as per my code understanding but not sure if I have missed anything...
          Hide
          Shalin Shekhar Mangar added a comment -

          Is there a specific reason why we have used "SegmentInfos.readCommit(...)" method instead of just using "commit.getFileNames()" ? It seems equivalent as per my code understanding but not sure if I have missed anything...

          I am not sure. Does commit.getFileNames() return the live files as well?

          Show
          Shalin Shekhar Mangar added a comment - Is there a specific reason why we have used "SegmentInfos.readCommit(...)" method instead of just using "commit.getFileNames()" ? It seems equivalent as per my code understanding but not sure if I have missed anything... I am not sure. Does commit.getFileNames() return the live files as well?

            People

            • Assignee:
              Varun Thacker
              Reporter:
              Varun Thacker
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development