Lucene - Core
  1. Lucene - Core
  2. LUCENE-5919

More carefully handle exceptions from IndexFileDeleter.decRef

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9.1, 4.10.1, 5.0, 6.0
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      From test failure: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/613/

      java.lang.AssertionError: TEST-TestIndexWriterOutOfMemory.testBasics-seed#[C4C9D7B4B9D5D290]: RefCount is 0 pre-decrement for file "_2_2_Lucene410_0.dvd"
      	at __randomizedtesting.SeedInfo.seed([C4C9D7B4B9D5D290:F9117998813B8CE0]:0)
      	at org.apache.lucene.index.IndexFileDeleter$RefCount.DecRef(IndexFileDeleter.java:725)
      	at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:611)
      	at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:599)
      	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2851)
      	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2952)
      	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2919)
      	at org.apache.lucene.index.TestIndexWriterOutOfMemory.testBasics(TestIndexWriterOutOfMemory.java:215)
      
      Reproduce with: ant test  -Dtestcase=TestIndexWriterOutOfMemory -Dtests.method=testBasics -Dtests.seed=C4C9D7B4B9D5D290 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=mk_MK -Dtests.timezone=America/Vancouver -Dtests.file.encoding=ISO-8859-1
      

      What happened was IW was trying to decRef a set of files, and at
      least one needed to be deleted, then virus checker threw exception,
      and IW later passed the full set of files back to decRef later despite
      that some had already been decRef'd the first time.

      1. LUCENE-5919.patch
        10 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch, I think it's ready.

        I fixed IFD.decRef to always complete its operation, but throw the first exception it hit (like IOUtils.close) and also added an IFD.decRefWhileHandlingException. I fixed IW to always "transactionally" fix its state when calling IFD.decRef even on exception ...

        Show
        Michael McCandless added a comment - Patch, I think it's ready. I fixed IFD.decRef to always complete its operation, but throw the first exception it hit (like IOUtils.close) and also added an IFD.decRefWhileHandlingException. I fixed IW to always "transactionally" fix its state when calling IFD.decRef even on exception ...
        Hide
        Robert Muir added a comment -

        +1, wow so many exc handling bugs here.

        Thanks Mike.

        Show
        Robert Muir added a comment - +1, wow so many exc handling bugs here. Thanks Mike.
        Hide
        ASF subversion and git services added a comment -

        Commit 1622279 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1622279 ]

        LUCENE-5919: fix IndexWriter exception handling when calling IndexFileDeleter.decRef

        Show
        ASF subversion and git services added a comment - Commit 1622279 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1622279 ] LUCENE-5919 : fix IndexWriter exception handling when calling IndexFileDeleter.decRef
        Hide
        ASF subversion and git services added a comment -

        Commit 1622281 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1622281 ]

        LUCENE-5919: fix IndexWriter exception handling when calling IndexFileDeleter.decRef

        Show
        ASF subversion and git services added a comment - Commit 1622281 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1622281 ] LUCENE-5919 : fix IndexWriter exception handling when calling IndexFileDeleter.decRef
        Hide
        Michael McCandless added a comment -

        I'd like to backport for 4.9.1 as well...

        Show
        Michael McCandless added a comment - I'd like to backport for 4.9.1 as well...
        Hide
        ASF subversion and git services added a comment -

        Commit 1625412 from Michael McCandless in branch 'dev/branches/lucene_solr_4_9'
        [ https://svn.apache.org/r1625412 ]

        LUCENE-5919: backport to 4.9.x

        Show
        ASF subversion and git services added a comment - Commit 1625412 from Michael McCandless in branch 'dev/branches/lucene_solr_4_9' [ https://svn.apache.org/r1625412 ] LUCENE-5919 : backport to 4.9.x
        Hide
        Michael McCandless added a comment -

        Reopen for backport to 4.10.1...

        Show
        Michael McCandless added a comment - Reopen for backport to 4.10.1...
        Hide
        ASF subversion and git services added a comment -

        Commit 1626187 from Robert Muir in branch 'dev/branches/lucene_solr_4_10'
        [ https://svn.apache.org/r1626187 ]

        LUCENE-5919: fix IndexWriter exception handling when calling IndexFileDeleter.decRef

        Show
        ASF subversion and git services added a comment - Commit 1626187 from Robert Muir in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1626187 ] LUCENE-5919 : fix IndexWriter exception handling when calling IndexFileDeleter.decRef
        Hide
        Michael McCandless added a comment -

        Bulk close for Lucene/Solr 4.10.1 release

        Show
        Michael McCandless added a comment - Bulk close for Lucene/Solr 4.10.1 release

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development