Solr
  1. Solr
  2. SOLR-4170

SOLR-4170: The 'backup' ReplicationHandler command can sometimes use a stale index directory rather than the current one.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0
    • Fix Version/s: 4.1, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      I'm using SolrCloud. When I'm trying to create index snapshot - exception occurs:
      INFO: [test] webapp=/solr path=/replication params=

      {command=backup&numberToKeep=1}

      status=0 QTime=1
      Dec 07, 2012 6:00:02 PM org.apache.solr.handler.SnapShooter createSnapshot
      SEVERE: Exception while creating snapshot
      java.io.FileNotFoundException: File /solr/cores/test/data/index/segments_g does not exist
      at org.apache.solr.handler.SnapShooter$FileCopier.copyFile(SnapShooter.java:194)
      at org.apache.solr.handler.SnapShooter$FileCopier.copyFiles(SnapShooter.java:185)
      at org.apache.solr.handler.SnapShooter.createSnapshot(SnapShooter.java:105)
      at org.apache.solr.handler.SnapShooter$1.run(SnapShooter.java:78)

      Issue occurs randomly. Reloading core usually helps, but sometimes core has to be reloaded couple of times to make snapshot possible.

        Activity

        Hide
        Tommaso Teofili added a comment - - edited

        where do you store index files? NFS?

        Show
        Tommaso Teofili added a comment - - edited where do you store index files? NFS?
        Hide
        Marcin Rzewucki added a comment -

        no, ephemeral storage in AWS (ext4)

        Show
        Marcin Rzewucki added a comment - no, ephemeral storage in AWS (ext4)
        Hide
        Marcin Rzewucki added a comment -

        The issue might be caused by frequent index changes. I found a temporary workaround for this: I stop updates, reload core on each node in SolrCloud (just for sure), create index snapshot and resume updates. Awkward, but works and I need reliable solution. Anyway, if solr has the ability to make an index snapshot it should take care of this process even in case of frequent updates. That's why I'd prefer to fix it in Solr instead of finding workarounds for this issue.

        Show
        Marcin Rzewucki added a comment - The issue might be caused by frequent index changes. I found a temporary workaround for this: I stop updates, reload core on each node in SolrCloud (just for sure), create index snapshot and resume updates. Awkward, but works and I need reliable solution. Anyway, if solr has the ability to make an index snapshot it should take care of this process even in case of frequent updates. That's why I'd prefer to fix it in Solr instead of finding workarounds for this issue.
        Hide
        Mark Miller added a comment -

        Any chance you can test the latest 4x build?

        Show
        Mark Miller added a comment - Any chance you can test the latest 4x build?
        Hide
        Mark Miller added a comment -

        I think this may be fixed on 4.X.

        Show
        Mark Miller added a comment - I think this may be fixed on 4.X.
        Hide
        Marcin Rzewucki added a comment -

        Tried with solr_41 (4.1.0.2012.12.14.09.42.29). It seems to be fixed only partially. Issue occurs once new index directory is created:
        SEVERE: Exception while creating snapshot
        java.io.FileNotFoundException: /solr/cores/aws/search/data/index/segments_1rg (No such file or directory)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(Unknown Source)
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:223)
        at org.apache.lucene.store.Directory.copy(Directory.java:200)
        at org.apache.solr.handler.SnapShooter$FileCopier.copyFile(SnapShooter.java:205)
        at org.apache.solr.handler.SnapShooter$FileCopier.copyFiles(SnapShooter.java:189)
        at org.apache.solr.handler.SnapShooter.createSnapshot(SnapShooter.java:107)
        at org.apache.solr.handler.SnapShooter$1.run(SnapShooter.java:77)

        File "segments_1rg" exists, but in /solr/cores/aws/search/data/index.20121227160834580/ directory. Seems like "index.properties"/getNewIndexDir is not respected by snapshotting process. Could you check that, please ?

        Show
        Marcin Rzewucki added a comment - Tried with solr_41 (4.1.0.2012.12.14.09.42.29). It seems to be fixed only partially. Issue occurs once new index directory is created: SEVERE: Exception while creating snapshot java.io.FileNotFoundException: /solr/cores/aws/search/data/index/segments_1rg (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(Unknown Source) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:223) at org.apache.lucene.store.Directory.copy(Directory.java:200) at org.apache.solr.handler.SnapShooter$FileCopier.copyFile(SnapShooter.java:205) at org.apache.solr.handler.SnapShooter$FileCopier.copyFiles(SnapShooter.java:189) at org.apache.solr.handler.SnapShooter.createSnapshot(SnapShooter.java:107) at org.apache.solr.handler.SnapShooter$1.run(SnapShooter.java:77) File "segments_1rg" exists, but in /solr/cores/aws/search/data/index.20121227160834580/ directory. Seems like "index.properties"/getNewIndexDir is not respected by snapshotting process. Could you check that, please ?
        Hide
        Mark Miller added a comment -

        Thanks for the report Marcin!

        Show
        Mark Miller added a comment - Thanks for the report Marcin!
        Hide
        Commit Tag Bot added a comment -

        [trunk commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1430111

        SOLR-4170: The 'backup' ReplicationHandler command can sometimes use a stale index directory rather than the current one.

        Show
        Commit Tag Bot added a comment - [trunk commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1430111 SOLR-4170 : The 'backup' ReplicationHandler command can sometimes use a stale index directory rather than the current one.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1430112

        SOLR-4170: The 'backup' ReplicationHandler command can sometimes use a stale index directory rather than the current one.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1430112 SOLR-4170 : The 'backup' ReplicationHandler command can sometimes use a stale index directory rather than the current one.

          People

          • Assignee:
            Mark Miller
            Reporter:
            Marcin Rzewucki
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development