Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7662

Index with missing files should throw CorruptIndexException

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.4
    • Fix Version/s: 7.0, 6.5
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Similar to what we did in LUCENE-7592 for EOF, we should catch missing files and rethrow those as CorruptIndexException.

      If a particular codec can handle missing files, it should be proactive check for those optional files and not throw anything, so I think we can safely do this at SegmentReader or SegmentCoreReaders level.

      Stack trace copied from SOLR-10006:

      Caused by: java.nio.file.NoSuchFileException: /Users/Erick/apache/solrVersions/trunk/solr/example/cloud/node3/solr/eoe_shard1_replica1/data/index/_1_Lucene50_0.doc
      	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
      	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
      	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
      	at java.nio.channels.FileChannel.open(FileChannel.java:287)
      	at java.nio.channels.FileChannel.open(FileChannel.java:335)
      	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
      	at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:192)
      	at org.apache.solr.core.MetricsDirectoryFactory$MetricsDirectory.openInput(MetricsDirectoryFactory.java:334)
      	at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.<init>(Lucene50PostingsReader.java:81)
      	at org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat.fieldsProducer(Lucene50PostingsFormat.java:442)
      	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:292)
      	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372)
      	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:109)
      	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
      	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
      	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:195)
      	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
      	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:473)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
      	at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
      	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1958)
      	... 12 more
      
      1. LUCENE-7662.patch
        9 kB
        Mike Drob
      2. LUCENE-7662.patch
        9 kB
        Mike Drob
      3. LUCENE-7662.patch
        6 kB
        Mike Drob

        Issue Links

          Activity

          Hide
          mikemccand Michael McCandless added a comment -

          +1

          Show
          mikemccand Michael McCandless added a comment - +1
          Hide
          mdrob Mike Drob added a comment -

          Michael McCandless - what do you think of this patch?

          Show
          mdrob Mike Drob added a comment - Michael McCandless - what do you think of this patch?
          Hide
          mikemccand Michael McCandless added a comment -

          Thanks Mike Drob; I think this patch looks good, except it makes some tests angry, e.g.:

             [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLucene62SegmentInfoFormat -Dtests.method=testRandomExceptions -Dtests.seed=F65CD1D4D104665D -Dtests.locale=zh -Dtests.timezone=Asia/Khandyga -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
             [junit4] ERROR   0.03s J3 | TestLucene62SegmentInfoFormat.testRandomExceptions <<<
             [junit4]    > Throwable #1: org.apache.lucene.index.CorruptIndexException: Problem reading index. (resource=a random IOException (_e.cfe))
             [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F65CD1D4D104665D:9E73BF104F0A3FFD]:0)
             [junit4]    > 	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:142)
             [junit4]    > 	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
             [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
             [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:195)
             [junit4]    > 	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103)
             [junit4]    > 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:473)
             [junit4]    > 	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103)
             [junit4]    > 	at org.apache.lucene.index.BaseIndexFileFormatTestCase.testRandomExceptions(BaseIndexFileFormatTestCase.java:563)
             [junit4]    > 	at org.apache.lucene.index.BaseSegmentInfoFormatTestCase.testRandomExceptions(BaseSegmentInfoFormatTestCase.java:50)
             [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
             [junit4]    > Caused by: java.nio.file.NoSuchFileException: a random IOException (_e.cfe)
             [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.maybeThrowIOExceptionOnOpen(MockDirectoryWrapper.java:575)
             [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:744)
             [junit4]    > 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
             [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
             [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:105)
             [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69)
             [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71)
             [junit4]    > 	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:99)
             [junit4]    > 	... 44 more
          

          Maybe we just need to relax that base test case to accept the new CorruptIndexExcpeption as well, and look to its cause to check the exception message?

          Also, I think it'd be a bit better to use our expectThrows method in the test case, wrapped around the one line where you try to open an index reader, instead of the @Test(expected = ...), which would pass if CorruptIndexException was hit anywhere in that test case?

          Show
          mikemccand Michael McCandless added a comment - Thanks Mike Drob ; I think this patch looks good, except it makes some tests angry, e.g.: [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestLucene62SegmentInfoFormat -Dtests.method=testRandomExceptions -Dtests.seed=F65CD1D4D104665D -Dtests.locale=zh -Dtests.timezone=Asia/Khandyga -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 0.03s J3 | TestLucene62SegmentInfoFormat.testRandomExceptions <<< [junit4] > Throwable #1: org.apache.lucene.index.CorruptIndexException: Problem reading index. (resource=a random IOException (_e.cfe)) [junit4] > at __randomizedtesting.SeedInfo.seed([F65CD1D4D104665D:9E73BF104F0A3FFD]:0) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:142) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143) [junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:195) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:103) [junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:473) [junit4] > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:103) [junit4] > at org.apache.lucene.index.BaseIndexFileFormatTestCase.testRandomExceptions(BaseIndexFileFormatTestCase.java:563) [junit4] > at org.apache.lucene.index.BaseSegmentInfoFormatTestCase.testRandomExceptions(BaseSegmentInfoFormatTestCase.java:50) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: java.nio.file.NoSuchFileException: a random IOException (_e.cfe) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.maybeThrowIOExceptionOnOpen(MockDirectoryWrapper.java:575) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:744) [junit4] > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:105) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:99) [junit4] > ... 44 more Maybe we just need to relax that base test case to accept the new CorruptIndexExcpeption as well, and look to its cause to check the exception message? Also, I think it'd be a bit better to use our expectThrows method in the test case, wrapped around the one line where you try to open an index reader, instead of the @Test(expected = ...) , which would pass if CorruptIndexException was hit anywhere in that test case?
          Hide
          mdrob Mike Drob added a comment -

          Those are good suggestions, I'll get them into the next version of this patch.

          Looking at the code in MockDirectoryWrapper, some of the "a random IOException" stuff looks really hackish, especially where we are checking for string messages to match. I'm uncomfortable with how brittle some of that is. We already have FakeIOException available and I think it would be good to use that instead in several places. Do you think we should handle that here, or I can file a new issue for it.

          Show
          mdrob Mike Drob added a comment - Those are good suggestions, I'll get them into the next version of this patch. Looking at the code in MockDirectoryWrapper, some of the "a random IOException" stuff looks really hackish, especially where we are checking for string messages to match. I'm uncomfortable with how brittle some of that is. We already have FakeIOException available and I think it would be good to use that instead in several places. Do you think we should handle that here, or I can file a new issue for it.
          Hide
          mdrob Mike Drob added a comment -

          Updated patch with some test clean up.

          Show
          mdrob Mike Drob added a comment - Updated patch with some test clean up.
          Hide
          mikemccand Michael McCandless added a comment -

          Thanks Mike Drob, the new patch looks great, and +1 to do that test cleanup here. I'll push soon!

          Show
          mikemccand Michael McCandless added a comment - Thanks Mike Drob , the new patch looks great, and +1 to do that test cleanup here. I'll push soon!
          Hide
          mikemccand Michael McCandless added a comment -

          Hmm something is still angry:

             [junit4] Suite: org.apache.lucene.index.TestMissingIndexFiles
             [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestMissingIndexFiles -Dtests.method=testMissingDoc -Dtests.seed=4D7CBCD6B337257 -Dtests.locale=de-CH -Dtests.timezone=Etc/GMT-10 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
             [junit4] FAILURE 0.04s J2 | TestMissingIndexFiles.testMissingDoc <<<
             [junit4]    > Throwable #1: junit.framework.AssertionFailedError: Expected exception CorruptIndexException
             [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4D7CBCD6B337257:6405AAA9369B3658]:0)
             [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2703)
             [junit4]    > 	at org.apache.lucene.index.TestMissingIndexFiles.testMissingDoc(TestMissingIndexFiles.java:52)
             [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
             [junit4]    > 	Suppressed: java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
             [junit4]    > 		at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:841)
             [junit4]    > 		at org.apache.lucene.index.TestMissingIndexFiles.testMissingDoc(TestMissingIndexFiles.java:53)
             [junit4]    > 		... 36 more
             [junit4]    > 	Caused by: java.lang.RuntimeException: unclosed IndexInput: _0.cfs
             [junit4]    > 		at org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:732)
             [junit4]    > 		at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:776)
             [junit4]    > 		at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:78)
             [junit4]    > 		at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71)
             [junit4]    > 		at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:99)
             [junit4]    > 		at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74)
             [junit4]    > 		at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
             [junit4]    > 		at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:54)
             [junit4]    > 		at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:666)
             [junit4]    > 		at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:77)
             [junit4]    > 		at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
             [junit4]    > 		at org.apache.lucene.index.TestMissingIndexFiles.lambda$testMissingDoc$0(TestMissingIndexFiles.java:52)
             [junit4]    > 		at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2694)
             [junit4]    > 		at org.apache.lucene.index.TestMissingIndexFiles.testMissingDoc(TestMissingIndexFiles.java:52)
             [junit4]    > 		... 36 more
          
          Show
          mikemccand Michael McCandless added a comment - Hmm something is still angry: [junit4] Suite: org.apache.lucene.index.TestMissingIndexFiles [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestMissingIndexFiles -Dtests.method=testMissingDoc -Dtests.seed=4D7CBCD6B337257 -Dtests.locale=de-CH -Dtests.timezone=Etc/GMT-10 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] FAILURE 0.04s J2 | TestMissingIndexFiles.testMissingDoc <<< [junit4] > Throwable #1: junit.framework.AssertionFailedError: Expected exception CorruptIndexException [junit4] > at __randomizedtesting.SeedInfo.seed([4D7CBCD6B337257:6405AAA9369B3658]:0) [junit4] > at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2703) [junit4] > at org.apache.lucene.index.TestMissingIndexFiles.testMissingDoc(TestMissingIndexFiles.java:52) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Suppressed: java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1} [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:841) [junit4] > at org.apache.lucene.index.TestMissingIndexFiles.testMissingDoc(TestMissingIndexFiles.java:53) [junit4] > ... 36 more [junit4] > Caused by: java.lang.RuntimeException: unclosed IndexInput: _0.cfs [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:732) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:776) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:78) [junit4] > at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) [junit4] > at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:99) [junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:74) [junit4] > at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62) [junit4] > at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:54) [junit4] > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:666) [junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:77) [junit4] > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) [junit4] > at org.apache.lucene.index.TestMissingIndexFiles.lambda$testMissingDoc$0(TestMissingIndexFiles.java:52) [junit4] > at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2694) [junit4] > at org.apache.lucene.index.TestMissingIndexFiles.testMissingDoc(TestMissingIndexFiles.java:52) [junit4] > ... 36 more
          Hide
          mdrob Mike Drob added a comment -

          Thanks. That is frustrating. I ran it 10 times and somehow never hit that or a similar seed. When the test uses the compound format, since there is no .doc file to remove, the index doesn't get corrupted and correctly never throws the exception. I couldn't figure out how to disable compound format from the test, so instead we can attempt to delete the doc or the .cfe file.

          I also made a change to check that we do delete something, otherwise the index would never be corrupt here. Since I can't imagine all possible future index file layouts, this seems prudent.

          Show
          mdrob Mike Drob added a comment - Thanks. That is frustrating. I ran it 10 times and somehow never hit that or a similar seed. When the test uses the compound format, since there is no .doc file to remove, the index doesn't get corrupted and correctly never throws the exception. I couldn't figure out how to disable compound format from the test, so instead we can attempt to delete the doc or the .cfe file. I also made a change to check that we do delete something, otherwise the index would never be corrupt here. Since I can't imagine all possible future index file layouts, this seems prudent.
          Hide
          mikemccand Michael McCandless added a comment -

          Thanks Mike Drob!

          That is frustrating. I ran it 10 times and somehow never hit that or a similar seed.

          The joys of randomized tests! I'll review and push soon...

          Show
          mikemccand Michael McCandless added a comment - Thanks Mike Drob ! That is frustrating. I ran it 10 times and somehow never hit that or a similar seed. The joys of randomized tests! I'll review and push soon...
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit be007a6c1124103ed4b21051a4ec98d94a2f7efa in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=be007a6 ]

          LUCENE-7662: throw CorruptIndexException if index files are missing

          Show
          jira-bot ASF subversion and git services added a comment - Commit be007a6c1124103ed4b21051a4ec98d94a2f7efa in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=be007a6 ] LUCENE-7662 : throw CorruptIndexException if index files are missing
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f8e16e3c4e852bce20d0793d157963777ac8f07b in lucene-solr's branch refs/heads/master from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f8e16e3 ]

          LUCENE-7662: move CHANGES entry under 6.5.0

          Show
          jira-bot ASF subversion and git services added a comment - Commit f8e16e3c4e852bce20d0793d157963777ac8f07b in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f8e16e3 ] LUCENE-7662 : move CHANGES entry under 6.5.0
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8ad763555012cd7637238380b6bfc5b4d7254325 in lucene-solr's branch refs/heads/branch_6x from Mike McCandless
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8ad7635 ]

          LUCENE-7662: throw CorruptIndexException if index files are missing

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8ad763555012cd7637238380b6bfc5b4d7254325 in lucene-solr's branch refs/heads/branch_6x from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8ad7635 ] LUCENE-7662 : throw CorruptIndexException if index files are missing
          Hide
          mikemccand Michael McCandless added a comment -

          Thank you Mike Drob!

          Show
          mikemccand Michael McCandless added a comment - Thank you Mike Drob !

            People

            • Assignee:
              Unassigned
              Reporter:
              mdrob Mike Drob
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development