Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.5
    • Fix Version/s: 4.9, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      It would be really nice if Solr supported named snapshots. Right now if you snapshot a SolrCloud cluster, every node potentially records a slightly different timestamp. Correlating those back together to effectively restore the entire cluster to a consistent snapshot is pretty tedious.

      1. SOLR-5340.patch
        19 kB
        Varun Thacker
      2. SOLR-5340.patch
        17 kB
        Varun Thacker
      3. SOLR-5340.patch
        17 kB
        Varun Thacker

        Issue Links

          Activity

          Hide
          Shalin Shekhar Mangar added a comment -

          Mike, this is good idea. If you can put up a patch, I'd be happy to review and commit it.

          Show
          Shalin Shekhar Mangar added a comment - Mike, this is good idea. If you can put up a patch, I'd be happy to review and commit it.
          Hide
          Varun Thacker added a comment -
          • Added the ability to create a named snapshot.
            Example - /replication?command=backup&name=testbackup
          • For named snapshots "maxNumberOfBackups" and "numberToKeep" are ignored.
          • Explicitly delete named snapshots
            Example - /replication?command=deletebackup&name=testbackup
          Show
          Varun Thacker added a comment - Added the ability to create a named snapshot. Example - /replication?command=backup&name=testbackup For named snapshots "maxNumberOfBackups" and "numberToKeep" are ignored. Explicitly delete named snapshots Example - /replication?command=deletebackup&name=testbackup
          Hide
          Noble Paul added a comment - - edited

          does it mean that a user will have to fire a request to all nodes where this collection is running? It is complex
          can this be a single collection admin command where you can say specify a collection name + snapshot name and the system can identify the nodes and fire separate requests to each node .

          I should be able to do the restore also similarly. Working with individual nodes should be discouraged as much as possible

          Show
          Noble Paul added a comment - - edited does it mean that a user will have to fire a request to all nodes where this collection is running? It is complex can this be a single collection admin command where you can say specify a collection name + snapshot name and the system can identify the nodes and fire separate requests to each node . I should be able to do the restore also similarly. Working with individual nodes should be discouraged as much as possible
          Hide
          Varun Thacker added a comment -

          I should have been more clear I guess. This was the approach I had planned to take -

          1. Use this Jira to add the ability for named snapshots/backups. This would be at a core level and thus could be used by non SolrCloud users also.
          2. In SOLR-5750 work on providing a seamless backup collection and restore collection API. it would utilise the work done on this Jira.

          Show
          Varun Thacker added a comment - I should have been more clear I guess. This was the approach I had planned to take - 1. Use this Jira to add the ability for named snapshots/backups. This would be at a core level and thus could be used by non SolrCloud users also. 2. In SOLR-5750 work on providing a seamless backup collection and restore collection API. it would utilise the work done on this Jira.
          Hide
          Noble Paul added a comment -

          OK, I'm not sure if SOLR-5750 would be able to use this . Because , you would want to save all data in a single place for a given collection and only a single copy (not one per replica) . To save something to a single location will be difficult using this API.

          My point is , I'm not sure if this will be directly re-usable for SOLR-5750 . But as a standalone feature this is a low hanging fruit and it should be fine and it does not have to be linked with SOLR-5750

          Show
          Noble Paul added a comment - OK, I'm not sure if SOLR-5750 would be able to use this . Because , you would want to save all data in a single place for a given collection and only a single copy (not one per replica) . To save something to a single location will be difficult using this API. My point is , I'm not sure if this will be directly re-usable for SOLR-5750 . But as a standalone feature this is a low hanging fruit and it should be fine and it does not have to be linked with SOLR-5750
          Hide
          Varun Thacker added a comment -

          Noble Paul This patch adds the ability to add a named snapshot at a specified location.

          When SOLR-5750 exposes a Collection API for BACKUP the "location" specified there would need to be a shared file system. Thus each shard would call it's ReplicationHandler and be able to backup it's entire index at the shared file system directory. It would also store cluster state etc. All that will be dealt in SOLR-5750.

          Show
          Varun Thacker added a comment - Noble Paul This patch adds the ability to add a named snapshot at a specified location. When SOLR-5750 exposes a Collection API for BACKUP the "location" specified there would need to be a shared file system. Thus each shard would call it's ReplicationHandler and be able to backup it's entire index at the shared file system directory. It would also store cluster state etc. All that will be dealt in SOLR-5750 .
          Hide
          Noble Paul added a comment -

          We need to do sufficient checks before performing "deletebackup" . In this patch , I can delete almost any directory in the server (or any location) as long as the process has sufficient permissions
          The snapshot name should have a well known pattern and check if the input is conforming to the pattern
          Also check if there is a real index in that location

          Show
          Noble Paul added a comment - We need to do sufficient checks before performing "deletebackup" . In this patch , I can delete almost any directory in the server (or any location) as long as the process has sufficient permissions The snapshot name should have a well known pattern and check if the input is conforming to the pattern Also check if there is a real index in that location
          Hide
          Varun Thacker added a comment -

          New patch with changes suggested by Noble.

          Show
          Varun Thacker added a comment - New patch with changes suggested by Noble.
          Hide
          Noble Paul added a comment -

          Sorry to raise this concern now. in deletebackup , Isn't possible to do the check of whether the snapshot exists etc to be done in the same thread and give a response back right away . It is much better than polling the status later

          I guess , even the backup command should do basic checks of the location etc before the call returns

          Show
          Noble Paul added a comment - Sorry to raise this concern now. in deletebackup , Isn't possible to do the check of whether the snapshot exists etc to be done in the same thread and give a response back right away . It is much better than polling the status later I guess , even the backup command should do basic checks of the location etc before the call returns
          Hide
          Varun Thacker added a comment -

          No problems Sorry I didn't get time to address it earlier.

          New patch which adds validateCreateSnapshot() and validateDeleteSnapshot() which throws SolrException if the validation fails. Once the validation succeeds the async requests for create and delete are made.

          Show
          Varun Thacker added a comment - No problems Sorry I didn't get time to address it earlier. New patch which adds validateCreateSnapshot() and validateDeleteSnapshot() which throws SolrException if the validation fails. Once the validation succeeds the async requests for create and delete are made.
          Hide
          ASF subversion and git services added a comment -

          Commit 1592957 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1592957 ]

          SOLR-5340: Add support for named snapshots

          Show
          ASF subversion and git services added a comment - Commit 1592957 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1592957 ] SOLR-5340 : Add support for named snapshots
          Hide
          ASF subversion and git services added a comment -

          Commit 1592958 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1592958 ]

          SOLR-5340: Add support for named snapshots

          Show
          ASF subversion and git services added a comment - Commit 1592958 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1592958 ] SOLR-5340 : Add support for named snapshots
          Hide
          Noble Paul added a comment -
          Show
          Noble Paul added a comment - Thanks Varun Thacker
          Hide
          Hoss Man added a comment -

          commit r1592957 added a "nocommit" to TestReplicationHandler

          Show
          Hoss Man added a comment - commit r1592957 added a "nocommit" to TestReplicationHandler
          Hide
          ASF subversion and git services added a comment -

          Commit 1593308 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1593308 ]

          SOLR-5340 removing a nocommit

          Show
          ASF subversion and git services added a comment - Commit 1593308 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1593308 ] SOLR-5340 removing a nocommit
          Hide
          ASF subversion and git services added a comment -

          Commit 1593314 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1593314 ]

          SOLR-5340 removing a nocommit

          Show
          ASF subversion and git services added a comment - Commit 1593314 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1593314 ] SOLR-5340 removing a nocommit
          Hide
          Mathias H. added a comment - - edited

          I think i found a bug (https://issues.apache.org/jira/browse/SOLR-6214) that is related with this feature.

          Show
          Mathias H. added a comment - - edited I think i found a bug ( https://issues.apache.org/jira/browse/SOLR-6214 ) that is related with this feature.
          Hide
          Noble Paul added a comment -

          The new issue is a bug so let us track it separately

          Show
          Noble Paul added a comment - The new issue is a bug so let us track it separately

            People

            • Assignee:
              Noble Paul
              Reporter:
              Mike Schrag
            • Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development