Lucene - Core
  1. Lucene - Core
  2. LUCENE-5904

Add MDW.enableVirusScanner / fix windows handling bugs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10.1, 5.0, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      IndexWriter has logic to handle the case where it can't delete a file (it puts in a retry list and indexfiledeleter will periodically retry, you can force this retry with deletePendingFiles).

      But from what I can tell, this logic is incomplete, e.g. its not properly handled during CFS creation, so if a file temporarily can't be deleted things like flush will fail.

      1. LUCENE-5904.patch
        86 kB
        Michael McCandless
      2. LUCENE-5904.patch
        91 kB
        Michael McCandless
      3. LUCENE-5904.patch
        37 kB
        Robert Muir
      4. LUCENE-5904.patch
        36 kB
        Robert Muir
      5. LUCENE-5904.patch
        36 kB
        Robert Muir
      6. LUCENE-5904.patch
        25 kB
        Robert Muir
      7. LUCENE-5904.patch
        1 kB
        Robert Muir

        Activity

        Hide
        Robert Muir added a comment -

        Here is a patch that just adds the logic to MDW. Some of the fails are false: e.g. tests directly against IFD or directory (These can just disable the option). But some, e.g. the CFS creation fails, are real.

        Show
        Robert Muir added a comment - Here is a patch that just adds the logic to MDW. Some of the fails are false: e.g. tests directly against IFD or directory (These can just disable the option). But some, e.g. the CFS creation fails, are real.
        Hide
        Robert Muir added a comment -

        Patch fixing 3 bugs so far (Lucene40SIWriter, Lucene46SIWriter, CompoundFileWriter). There might be more bugs: we should review all uses of Directory.deleteFile to make sure we are doing the right thing.

        I also fixed up core tests that currently rely upon e.g. unref'ed files check or manipulate files directly to disable the option.

        I may have made a mistake or unconvered something in disk full test that i havent investigated yet:

           [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull
           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterOnDiskFull -Dtests.method=testImmediateDiskFull -Dtests.seed=2D75D397EE0B3214 -Dtests.locale=ga_IE -Dtests.timezone=Europe/Lisbon -Dtests.file.encoding=ISO-8859-1
           [junit4] ERROR   0.20s | TestIndexWriterOnDiskFull.testImmediateDiskFull <<<
           [junit4]    > Throwable #1: java.io.EOFException: read past EOF: RAMInputStream(name=segments_1)
           [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2D75D397EE0B3214:BC3341FB920ADCD0]:0)
           [junit4]    > 	at org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:98)
           [junit4]    > 	at org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:71)
           [junit4]    > 	at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:122)
           [junit4]    > 	at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41)
           [junit4]    > 	at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:356)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:463)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:804)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:650)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:459)
           [junit4]    > 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:398)
           [junit4]    > 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:207)
           [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:711)
           [junit4]    > 	at org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:569)
           [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
        

        Maybe something isn't quite right in the windows handlign there?

        Show
        Robert Muir added a comment - Patch fixing 3 bugs so far (Lucene40SIWriter, Lucene46SIWriter, CompoundFileWriter). There might be more bugs: we should review all uses of Directory.deleteFile to make sure we are doing the right thing. I also fixed up core tests that currently rely upon e.g. unref'ed files check or manipulate files directly to disable the option. I may have made a mistake or unconvered something in disk full test that i havent investigated yet: [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterOnDiskFull -Dtests.method=testImmediateDiskFull -Dtests.seed=2D75D397EE0B3214 -Dtests.locale=ga_IE -Dtests.timezone=Europe/Lisbon -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 0.20s | TestIndexWriterOnDiskFull.testImmediateDiskFull <<< [junit4] > Throwable #1: java.io.EOFException: read past EOF: RAMInputStream(name=segments_1) [junit4] > at __randomizedtesting.SeedInfo.seed([2D75D397EE0B3214:BC3341FB920ADCD0]:0) [junit4] > at org.apache.lucene.store.RAMInputStream.switchCurrentBuffer(RAMInputStream.java:98) [junit4] > at org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:71) [junit4] > at org.apache.lucene.store.MockIndexInputWrapper.readByte(MockIndexInputWrapper.java:122) [junit4] > at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) [junit4] > at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) [junit4] > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:356) [junit4] > at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:463) [junit4] > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:804) [junit4] > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:650) [junit4] > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:459) [junit4] > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:398) [junit4] > at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:207) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:711) [junit4] > at org.apache.lucene.index.TestIndexWriterOnDiskFull.testImmediateDiskFull(TestIndexWriterOnDiskFull.java:569) [junit4] > at java.lang.Thread.run(Thread.java:745) Maybe something isn't quite right in the windows handlign there?
        Hide
        Uwe Schindler added a comment -

        I like the virus scanner, just doing nothing - only holding files open I wish all virus scanner would do nothing!

        Show
        Uwe Schindler added a comment - I like the virus scanner, just doing nothing - only holding files open I wish all virus scanner would do nothing!
        Hide
        Michael McCandless added a comment -

        +1, this is a nice new evilness for MDW!

        But that EOFE is terrifying?

        Show
        Michael McCandless added a comment - +1, this is a nice new evilness for MDW! But that EOFE is terrifying?
        Hide
        Robert Muir added a comment -

        If its because it writes incomplete commit and won't fall back to "nothing", maybe instead of weakening the test we can just do an empty commit up front so we still keep coverage.

        Its worth it for the disk full tests, i dont really want them lenient.

        Show
        Robert Muir added a comment - If its because it writes incomplete commit and won't fall back to "nothing", maybe instead of weakening the test we can just do an empty commit up front so we still keep coverage. Its worth it for the disk full tests, i dont really want them lenient.
        Hide
        Michael McCandless added a comment -

        OK the EOFE is "just" because the very first commit is corrupt.

        It happens with this seed because MDW throws an exc when IW is writing segments_1, and then IW tries to remove segments_1 and MDW throws another exception (new virus checker in this patch) and so a corrupt segments_1 is left. If there were a prior commit, then at read time we would fall back to it.

        So net/net I don't think there's anything to fix here, except +1 to just have the test make an empty first commit (before any MDW exceptions are enabled).

        Show
        Michael McCandless added a comment - OK the EOFE is "just" because the very first commit is corrupt. It happens with this seed because MDW throws an exc when IW is writing segments_1, and then IW tries to remove segments_1 and MDW throws another exception (new virus checker in this patch) and so a corrupt segments_1 is left. If there were a prior commit, then at read time we would fall back to it. So net/net I don't think there's anything to fix here, except +1 to just have the test make an empty first commit (before any MDW exceptions are enabled).
        Hide
        Robert Muir added a comment -

        Updated patch. I also fixed a few more false fails.

        Still in general, there are interesting failures every time you run core tests with the patch. testThreadInterruptDeadlock got angry because write.lock couldn't be removed, need to investigate that deletion further.

        I also havent looked at this:

           [junit4] Suite: org.apache.lucene.index.TestCodecHoldsOpenFiles
           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecHoldsOpenFiles -Dtests.method=test -Dtests.seed=1908AF7C5FA5D64A -Dtests.locale=sl -Dtests.timezone=Asia/Bangkok -Dtests.file.encoding=ISO-8859-1
           [junit4] ERROR   0.01s J1 | TestCodecHoldsOpenFiles.test <<<
           [junit4]    > Throwable #1: java.io.FileNotFoundException: segments_1 in dir=RAMDirectory@25ac448 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@150c542d
           [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1908AF7C5FA5D64A:915C90A6F159BBB2]:0)
           [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:593)
           [junit4]    > 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:106)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:347)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:458)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:794)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:640)
           [junit4]    > 	at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:454)
           [junit4]    > 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:398)
        
        Show
        Robert Muir added a comment - Updated patch. I also fixed a few more false fails. Still in general, there are interesting failures every time you run core tests with the patch. testThreadInterruptDeadlock got angry because write.lock couldn't be removed, need to investigate that deletion further. I also havent looked at this: [junit4] Suite: org.apache.lucene.index.TestCodecHoldsOpenFiles [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestCodecHoldsOpenFiles -Dtests.method=test -Dtests.seed=1908AF7C5FA5D64A -Dtests.locale=sl -Dtests.timezone=Asia/Bangkok -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 0.01s J1 | TestCodecHoldsOpenFiles.test <<< [junit4] > Throwable #1: java.io.FileNotFoundException: segments_1 in dir=RAMDirectory@25ac448 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@150c542d [junit4] > at __randomizedtesting.SeedInfo.seed([1908AF7C5FA5D64A:915C90A6F159BBB2]:0) [junit4] > at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:593) [junit4] > at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:106) [junit4] > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:347) [junit4] > at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:458) [junit4] > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:794) [junit4] > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:640) [junit4] > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:454) [junit4] > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:398)
        Hide
        Robert Muir added a comment -

        I think its just the long tail left now:

        • testThreadInterruptDeadLock and write.lock
        • testCodecHoldsOpenFiles and whatever is going on there
        • replicator tests (i dont know if its just a test issue and virus scanner should be disabled)
        Show
        Robert Muir added a comment - I think its just the long tail left now: testThreadInterruptDeadLock and write.lock testCodecHoldsOpenFiles and whatever is going on there replicator tests (i dont know if its just a test issue and virus scanner should be disabled)
        Hide
        Robert Muir added a comment -

        I fixed the CodecHoldsOpenFiles test. It was buggy before: it might not even ever commit due to close randomization, and the checkIndex at the end would just do nothing.

        Show
        Robert Muir added a comment - I fixed the CodecHoldsOpenFiles test. It was buggy before: it might not even ever commit due to close randomization, and the checkIndex at the end would just do nothing.
        Hide
        ASF subversion and git services added a comment -

        Commit 1620340 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620340 ]

        LUCENE-5904: make branch

        Show
        ASF subversion and git services added a comment - Commit 1620340 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620340 ] LUCENE-5904 : make branch
        Hide
        ASF subversion and git services added a comment -

        Commit 1620341 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620341 ]

        LUCENE-5904: current patch

        Show
        ASF subversion and git services added a comment - Commit 1620341 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620341 ] LUCENE-5904 : current patch
        Hide
        ASF subversion and git services added a comment -

        Commit 1620342 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620342 ]

        LUCENE-5904: MDW confesses when virus checker kicks in, if you run verbose

        Show
        ASF subversion and git services added a comment - Commit 1620342 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620342 ] LUCENE-5904 : MDW confesses when virus checker kicks in, if you run verbose
        Hide
        ASF subversion and git services added a comment -

        Commit 1620343 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620343 ]

        LUCENE-5904: fix false test failure

        Show
        ASF subversion and git services added a comment - Commit 1620343 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620343 ] LUCENE-5904 : fix false test failure
        Hide
        ASF subversion and git services added a comment -

        Commit 1620418 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620418 ]

        LUCENE-5904: fix false fails

        Show
        ASF subversion and git services added a comment - Commit 1620418 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620418 ] LUCENE-5904 : fix false fails
        Hide
        ASF subversion and git services added a comment -

        Commit 1620421 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620421 ]

        LUCENE-5904: fix false fail

        Show
        ASF subversion and git services added a comment - Commit 1620421 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620421 ] LUCENE-5904 : fix false fail
        Hide
        Robert Muir added a comment -

        Another real bug i think, in IndexFileDeleter (found by TestCrash)

        IW crashes or something, and we have some leftover files (like _0.si, imagine from an initial empty commit).

        when we bootup a new IW, it tries to delete the trash, but for some reason temporarily cannot delete _0.si. Then we go and flush real segment _0, only afterwards IFD comes back around and deletes _0.si, which is now a legit file, corrupting the index.

        Its caused by the filename reuse problem (LUCENE-5903).

        Show
        Robert Muir added a comment - Another real bug i think, in IndexFileDeleter (found by TestCrash) IW crashes or something, and we have some leftover files (like _0.si, imagine from an initial empty commit). when we bootup a new IW, it tries to delete the trash, but for some reason temporarily cannot delete _0.si. Then we go and flush real segment _0, only afterwards IFD comes back around and deletes _0.si, which is now a legit file, corrupting the index. Its caused by the filename reuse problem ( LUCENE-5903 ).
        Hide
        ASF subversion and git services added a comment -

        Commit 1620451 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620451 ]

        LUCENE-5904: improve debuggability on fail

        Show
        ASF subversion and git services added a comment - Commit 1620451 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620451 ] LUCENE-5904 : improve debuggability on fail
        Hide
        ASF subversion and git services added a comment -

        Commit 1620575 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620575 ]

        LUCENE-5904: fix replicator tests

        Show
        ASF subversion and git services added a comment - Commit 1620575 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620575 ] LUCENE-5904 : fix replicator tests
        Hide
        ASF subversion and git services added a comment -

        Commit 1620576 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620576 ]

        LUCENE-5904: fix false failures

        Show
        ASF subversion and git services added a comment - Commit 1620576 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620576 ] LUCENE-5904 : fix false failures
        Hide
        ASF subversion and git services added a comment -

        Commit 1620580 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620580 ]

        LUCENE-5904: fix fails

        Show
        ASF subversion and git services added a comment - Commit 1620580 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620580 ] LUCENE-5904 : fix fails
        Hide
        ASF subversion and git services added a comment -

        Commit 1620582 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620582 ]

        LUCENE-5904: fix false fails

        Show
        ASF subversion and git services added a comment - Commit 1620582 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620582 ] LUCENE-5904 : fix false fails
        Hide
        ASF subversion and git services added a comment -

        Commit 1620584 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620584 ]

        LUCENE-5904: fix false fail

        Show
        ASF subversion and git services added a comment - Commit 1620584 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620584 ] LUCENE-5904 : fix false fail
        Hide
        ASF subversion and git services added a comment -

        Commit 1620601 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620601 ]

        LUCENE-5904: add explicit test

        Show
        ASF subversion and git services added a comment - Commit 1620601 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620601 ] LUCENE-5904 : add explicit test
        Hide
        ASF subversion and git services added a comment -

        Commit 1620753 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620753 ]

        LUCENE-5904: first cut at gen inflation to prevent index corruption when files are re-used after first IW has unclean shutdown and 2nd IW encounters virus checker

        Show
        ASF subversion and git services added a comment - Commit 1620753 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620753 ] LUCENE-5904 : first cut at gen inflation to prevent index corruption when files are re-used after first IW has unclean shutdown and 2nd IW encounters virus checker
        Hide
        ASF subversion and git services added a comment -

        Commit 1620755 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620755 ]

        LUCENE-5904: set MDW back to try to provoke more fails

        Show
        ASF subversion and git services added a comment - Commit 1620755 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620755 ] LUCENE-5904 : set MDW back to try to provoke more fails
        Hide
        ASF subversion and git services added a comment -

        Commit 1620756 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620756 ]

        LUCENE-5904: remove sop

        Show
        ASF subversion and git services added a comment - Commit 1620756 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620756 ] LUCENE-5904 : remove sop
        Hide
        ASF subversion and git services added a comment -

        Commit 1620777 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620777 ]

        LUCENE-5904: don't parse segments.gen as a segments file and overinflate

        Show
        ASF subversion and git services added a comment - Commit 1620777 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620777 ] LUCENE-5904 : don't parse segments.gen as a segments file and overinflate
        Hide
        ASF subversion and git services added a comment -

        Commit 1620778 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620778 ]

        LUCENE-5904: test segments inflation

        Show
        ASF subversion and git services added a comment - Commit 1620778 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620778 ] LUCENE-5904 : test segments inflation
        Hide
        ASF subversion and git services added a comment -

        Commit 1620779 from Robert Muir in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620779 ]

        LUCENE-5904: add tests for segment/gen inflation, fix gen inflation (it always parsed gen of 0), fix off-by-one in gen-inflation (was starting at 2 instead of 1), add robustness to trash files we might be looking at, add minor restriction to segment suffix names so we can always parse generations correctly from them

        Show
        ASF subversion and git services added a comment - Commit 1620779 from Robert Muir in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620779 ] LUCENE-5904 : add tests for segment/gen inflation, fix gen inflation (it always parsed gen of 0), fix off-by-one in gen-inflation (was starting at 2 instead of 1), add robustness to trash files we might be looking at, add minor restriction to segment suffix names so we can always parse generations correctly from them
        Hide
        ASF subversion and git services added a comment -

        Commit 1620802 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620802 ]

        LUCENE-5904: address last nocommit

        Show
        ASF subversion and git services added a comment - Commit 1620802 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620802 ] LUCENE-5904 : address last nocommit
        Hide
        ASF subversion and git services added a comment -

        Commit 1620820 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620820 ]

        LUCENE-5904: fix some false false

        Show
        ASF subversion and git services added a comment - Commit 1620820 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620820 ] LUCENE-5904 : fix some false false
        Hide
        ASF subversion and git services added a comment -

        Commit 1620852 from Michael McCandless in branch 'dev/branches/lucene5904'
        [ https://svn.apache.org/r1620852 ]

        LUCENE-5904: merge trunk

        Show
        ASF subversion and git services added a comment - Commit 1620852 from Michael McCandless in branch 'dev/branches/lucene5904' [ https://svn.apache.org/r1620852 ] LUCENE-5904 : merge trunk
        Hide
        Michael McCandless added a comment -

        Patch from diffSources (trunk vs branch); I think it's ready. I ran 52 iters of distributed beasting (all lucene core + modules tests) ...

        Show
        Michael McCandless added a comment - Patch from diffSources (trunk vs branch); I think it's ready. I ran 52 iters of distributed beasting (all lucene core + modules tests) ...
        Hide
        Michael McCandless added a comment -

        To summarize this issue:

        First Rob added nice new evilness to MDW so that sometimes Directory.deleteFile would fail even if Lucene did not have that file open, simulating a virus checker temporarily holding the file open. (Previously this operation would always succeed).

        But this new evilness uncovered a nasty corruption case in Lucene, whereby 1) an unclean shutdown of a previous IW left some "future" segment files in the index, e.g. _5.pos, 2) the new IW starts up and identifies this file as not being referenced and immediately tries to delete it, but 3) the virus checker prevents _5.pos being deleted on init. Normally this is "ok": IW records that this file needs deleting but failed last time and so it periodically retries.

        The problem is, when the IW goes and flushes a few segments, it may now in fact overwrite _5.pos with a "real" one, which may succeed (if virus checker is done with that file), and then later when IW retries its deletes, it removes _5.pos, corrupting the index.

        I don't know of any actual user cases showing this corruption case ... but it's quite insidious ...

        Show
        Michael McCandless added a comment - To summarize this issue: First Rob added nice new evilness to MDW so that sometimes Directory.deleteFile would fail even if Lucene did not have that file open, simulating a virus checker temporarily holding the file open. (Previously this operation would always succeed). But this new evilness uncovered a nasty corruption case in Lucene, whereby 1) an unclean shutdown of a previous IW left some "future" segment files in the index, e.g. _5.pos, 2) the new IW starts up and identifies this file as not being referenced and immediately tries to delete it, but 3) the virus checker prevents _5.pos being deleted on init. Normally this is "ok": IW records that this file needs deleting but failed last time and so it periodically retries. The problem is, when the IW goes and flushes a few segments, it may now in fact overwrite _5.pos with a "real" one, which may succeed (if virus checker is done with that file), and then later when IW retries its deletes, it removes _5.pos, corrupting the index. I don't know of any actual user cases showing this corruption case ... but it's quite insidious ...
        Hide
        Michael McCandless added a comment -

        New patch, just adding a check & assert in IFD that it should never delete a pending file that has a non-zero refCount.

        I think it's ready ... I'll commit later today ...

        Show
        Michael McCandless added a comment - New patch, just adding a check & assert in IFD that it should never delete a pending file that has a non-zero refCount. I think it's ready ... I'll commit later today ...
        Hide
        ASF subversion and git services added a comment -

        Commit 1621389 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1621389 ]

        LUCENE-5904: fix corruption case caused by virus checker after an unclean IW shutdown

        Show
        ASF subversion and git services added a comment - Commit 1621389 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1621389 ] LUCENE-5904 : fix corruption case caused by virus checker after an unclean IW shutdown
        Hide
        ASF subversion and git services added a comment -

        Commit 1621392 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1621392 ]

        LUCENE-5904: fix corruption case caused by virus checker after an unclean IW shutdown

        Show
        ASF subversion and git services added a comment - Commit 1621392 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1621392 ] LUCENE-5904 : fix corruption case caused by virus checker after an unclean IW shutdown
        Hide
        ASF subversion and git services added a comment -

        Commit 1621421 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1621421 ]

        LUCENE-5904: fix test to make empty initial commit

        Show
        ASF subversion and git services added a comment - Commit 1621421 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1621421 ] LUCENE-5904 : fix test to make empty initial commit
        Hide
        ASF subversion and git services added a comment -

        Commit 1621422 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1621422 ]

        LUCENE-5904: fix test to make empty initial commit

        Show
        ASF subversion and git services added a comment - Commit 1621422 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1621422 ] LUCENE-5904 : fix test to make empty initial commit
        Hide
        ASF subversion and git services added a comment -

        Commit 1621423 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1621423 ]

        LUCENE-5904: fix redundant cast warning

        Show
        ASF subversion and git services added a comment - Commit 1621423 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1621423 ] LUCENE-5904 : fix redundant cast warning
        Hide
        ASF subversion and git services added a comment -

        Commit 1621424 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1621424 ]

        LUCENE-5904: fix redundant cast warning

        Show
        ASF subversion and git services added a comment - Commit 1621424 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1621424 ] LUCENE-5904 : fix redundant cast warning
        Hide
        ASF subversion and git services added a comment -

        Commit 1621428 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1621428 ]

        LUCENE-5904: fix false fail

        Show
        ASF subversion and git services added a comment - Commit 1621428 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1621428 ] LUCENE-5904 : fix false fail
        Hide
        ASF subversion and git services added a comment -

        Commit 1621429 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1621429 ]

        LUCENE-5904: fix false fail

        Show
        ASF subversion and git services added a comment - Commit 1621429 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1621429 ] LUCENE-5904 : fix false fail
        Hide
        Michael McCandless added a comment -

        Reopen to backport to 4.10.1

        Show
        Michael McCandless added a comment - Reopen to backport to 4.10.1
        Hide
        ASF subversion and git services added a comment -

        Commit 1626204 from Michael McCandless in branch 'dev/branches/lucene_solr_4_10'
        [ https://svn.apache.org/r1626204 ]

        LUCENE-5904: backport to 4.10.1

        Show
        ASF subversion and git services added a comment - Commit 1626204 from Michael McCandless in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1626204 ] LUCENE-5904 : backport to 4.10.1
        Hide
        Michael McCandless added a comment -

        Bulk close for Lucene/Solr 4.10.1 release

        Show
        Michael McCandless added a comment - Bulk close for Lucene/Solr 4.10.1 release

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development