Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5919

More carefully handle exceptions from IndexFileDeleter.decRef

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9.1, 4.10.1, 5.0, 6.0
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      From test failure: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/613/

      java.lang.AssertionError: TEST-TestIndexWriterOutOfMemory.testBasics-seed#[C4C9D7B4B9D5D290]: RefCount is 0 pre-decrement for file "_2_2_Lucene410_0.dvd"
      	at __randomizedtesting.SeedInfo.seed([C4C9D7B4B9D5D290:F9117998813B8CE0]:0)
      	at org.apache.lucene.index.IndexFileDeleter$RefCount.DecRef(IndexFileDeleter.java:725)
      	at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:611)
      	at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:599)
      	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2851)
      	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2952)
      	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2919)
      	at org.apache.lucene.index.TestIndexWriterOutOfMemory.testBasics(TestIndexWriterOutOfMemory.java:215)
      
      Reproduce with: ant test  -Dtestcase=TestIndexWriterOutOfMemory -Dtests.method=testBasics -Dtests.seed=C4C9D7B4B9D5D290 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=mk_MK -Dtests.timezone=America/Vancouver -Dtests.file.encoding=ISO-8859-1
      

      What happened was IW was trying to decRef a set of files, and at
      least one needed to be deleted, then virus checker threw exception,
      and IW later passed the full set of files back to decRef later despite
      that some had already been decRef'd the first time.

      1. LUCENE-5919.patch
        10 kB
        Michael McCandless

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        Patch, I think it's ready.

        I fixed IFD.decRef to always complete its operation, but throw the first exception it hit (like IOUtils.close) and also added an IFD.decRefWhileHandlingException. I fixed IW to always "transactionally" fix its state when calling IFD.decRef even on exception ...

        Show
        mikemccand Michael McCandless added a comment - Patch, I think it's ready. I fixed IFD.decRef to always complete its operation, but throw the first exception it hit (like IOUtils.close) and also added an IFD.decRefWhileHandlingException. I fixed IW to always "transactionally" fix its state when calling IFD.decRef even on exception ...
        Hide
        rcmuir Robert Muir added a comment -

        +1, wow so many exc handling bugs here.

        Thanks Mike.

        Show
        rcmuir Robert Muir added a comment - +1, wow so many exc handling bugs here. Thanks Mike.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1622279 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1622279 ]

        LUCENE-5919: fix IndexWriter exception handling when calling IndexFileDeleter.decRef

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1622279 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1622279 ] LUCENE-5919 : fix IndexWriter exception handling when calling IndexFileDeleter.decRef
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1622281 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1622281 ]

        LUCENE-5919: fix IndexWriter exception handling when calling IndexFileDeleter.decRef

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1622281 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1622281 ] LUCENE-5919 : fix IndexWriter exception handling when calling IndexFileDeleter.decRef
        Hide
        mikemccand Michael McCandless added a comment -

        I'd like to backport for 4.9.1 as well...

        Show
        mikemccand Michael McCandless added a comment - I'd like to backport for 4.9.1 as well...
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1625412 from Michael McCandless in branch 'dev/branches/lucene_solr_4_9'
        [ https://svn.apache.org/r1625412 ]

        LUCENE-5919: backport to 4.9.x

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1625412 from Michael McCandless in branch 'dev/branches/lucene_solr_4_9' [ https://svn.apache.org/r1625412 ] LUCENE-5919 : backport to 4.9.x
        Hide
        mikemccand Michael McCandless added a comment -

        Reopen for backport to 4.10.1...

        Show
        mikemccand Michael McCandless added a comment - Reopen for backport to 4.10.1...
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1626187 from Robert Muir in branch 'dev/branches/lucene_solr_4_10'
        [ https://svn.apache.org/r1626187 ]

        LUCENE-5919: fix IndexWriter exception handling when calling IndexFileDeleter.decRef

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1626187 from Robert Muir in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1626187 ] LUCENE-5919 : fix IndexWriter exception handling when calling IndexFileDeleter.decRef
        Hide
        mikemccand Michael McCandless added a comment -

        Bulk close for Lucene/Solr 4.10.1 release

        Show
        mikemccand Michael McCandless added a comment - Bulk close for Lucene/Solr 4.10.1 release

          People

          • Assignee:
            mikemccand Michael McCandless
            Reporter:
            mikemccand Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development