Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6762

CheckIndex cannot "fix" indexes that have individual segments with missing or corrupt .si files because sanity checks will fail trying to read the index initially.

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Seems like we should still be able to partially recover by dropping these segments with CheckIndex.

        Issue Links

          Activity

          Hide
          mikemccand Michael McCandless added a comment -

          Perhaps it is better to catch all of IOException on that read?

          I think that makes sense, but it's maybe a bit spooky, e.g. it could include transient IO issues like too-many-open-files, but again so would the rest of the code in CheckIndex ... so I guess that makes sense?

          Show
          mikemccand Michael McCandless added a comment - Perhaps it is better to catch all of IOException on that read? I think that makes sense, but it's maybe a bit spooky, e.g. it could include transient IO issues like too-many-open-files, but again so would the rest of the code in CheckIndex ... so I guess that makes sense?
          Hide
          mdrob Mike Drob added a comment -

          I noticed that in LUCENE-5947 Taiki got an java.io.EOFException on the missing file. Perhaps it is better to catch all of IOException on that read?

          Show
          mdrob Mike Drob added a comment - I noticed that in LUCENE-5947 Taiki got an java.io.EOFException on the missing file. Perhaps it is better to catch all of IOException on that read?
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Note to give Taiki Sugawara credit for the original bug report on this in LUCENE-5947.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Note to give Taiki Sugawara credit for the original bug report on this in LUCENE-5947 .
          Hide
          mikemccand Michael McCandless added a comment -

          do you think there is additional value in the full trace?

          I think possibly? E.g. it shows which codec (well, SegmentInfoFormat) is being used to load the SI, if it's a crazy format that needs 3 separate files to write the SegmentInfo, we'd see which file was missing, etc. Plus the rest of CheckIndex generally shows the full exception of "what went wrong"...

          Show
          mikemccand Michael McCandless added a comment - do you think there is additional value in the full trace? I think possibly? E.g. it shows which codec (well, SegmentInfoFormat) is being used to load the SI, if it's a crazy format that needs 3 separate files to write the SegmentInfo, we'd see which file was missing, etc. Plus the rest of CheckIndex generally shows the full exception of "what went wrong"...
          Hide
          mdrob Mike Drob added a comment -

          I'll make the methods package-private and add NoSuchFileException.

          If the user specifies verbose option, then I added that we pass the infoStream to SegmentInfos from CheckIndex. Current behaviour is to print just the missing segment name - do you think there is additional value in the full trace?

          Show
          mdrob Mike Drob added a comment - I'll make the methods package-private and add NoSuchFileException . If the user specifies verbose option, then I added that we pass the infoStream to SegmentInfos from CheckIndex . Current behaviour is to print just the missing segment name - do you think there is additional value in the full trace?
          Hide
          mikemccand Michael McCandless added a comment -

          skipMissing is a spooky option; if accidentally mis-used it can make you think you have all segments if you forget to check the missingSegmentInfo ... can we make this new API package private and leave the public API unchanged?

          Shouldn't we also catch NoSuchFileException?

          Can you have the precise exception that was hit, and print it to the user from CheckIndex?

          Show
          mikemccand Michael McCandless added a comment - skipMissing is a spooky option; if accidentally mis-used it can make you think you have all segments if you forget to check the missingSegmentInfo ... can we make this new API package private and leave the public API unchanged? Shouldn't we also catch NoSuchFileException ? Can you have the precise exception that was hit, and print it to the user from CheckIndex ?
          Hide
          mikemccand Michael McCandless added a comment -

          LUCENE-5947 is a dup of this issue ...

          Show
          mikemccand Michael McCandless added a comment - LUCENE-5947 is a dup of this issue ...
          Hide
          mdrob Mike Drob added a comment -

          This is a moderately invasive patch into SegmentInfos, so please review carefully. Adds an option to skip missing segments (currently only used by CheckIndex)and a couple of additional printed messages relating to this error.

          Show
          mdrob Mike Drob added a comment - This is a moderately invasive patch into SegmentInfos, so please review carefully. Adds an option to skip missing segments (currently only used by CheckIndex)and a couple of additional printed messages relating to this error.
          Hide
          mdrob Mike Drob added a comment -

          I've almost got a patch ready for this, probably will finish it up tomorrow.

          Show
          mdrob Mike Drob added a comment - I've almost got a patch ready for this, probably will finish it up tomorrow.
          Hide
          mdrob Mike Drob added a comment -

          A stack trace representative of the issue (this is from LUCENE 4.4)

          java.io.FileNotFoundException: /path/to/index/_879r.si (No such file or directory)
                  at java.io.RandomAccessFile.open(Native Method)
                  at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
                  at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:193)
                  at org.apache.lucene.codecs.lucene40.Lucene40SegmentInfoReader.read(Lucene40SegmentInfoReader.java:50)
                  at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:334)
                  at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:380)
                  at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:812)
                  at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:663)
                  at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:376)
                  at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:382)
                  at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1854)
          

          You can fake out the sanity checks by copying some other *.si file in place, and initial impression is that the tool works.

          Show
          mdrob Mike Drob added a comment - A stack trace representative of the issue (this is from LUCENE 4.4) java.io.FileNotFoundException: /path/to/index/_879r.si (No such file or directory) at java.io.RandomAccessFile.open(Native Method)         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)         at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:193)         at org.apache.lucene.codecs.lucene40.Lucene40SegmentInfoReader.read(Lucene40SegmentInfoReader.java:50)         at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:334)         at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:380)         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:812)         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:663)         at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:376) at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:382)         at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1854) You can fake out the sanity checks by copying some other *.si file in place, and initial impression is that the tool works.

            People

            • Assignee:
              Unassigned
              Reporter:
              markrmiller@gmail.com Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development