Lucene - Core
  1. Lucene - Core
  2. LUCENE-5330

IndexWriter doesn't process all events on getReader / close / rollback

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.5, 6.0
    • Fix Version/s: 4.6, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      IndexWriter misses to apply all pending events in getReader() as well as close() / rollback(). This can lead to files that never get deleted or only very very late. If you are using RAM Directories for instance this quickly fills up a huge amount of memory. It might not be super critical in production and it also doesn't cause any data loss or index corruption.

      1. LUCENE-5330.patch
        2 kB
        Simon Willnauer

        Activity

        Hide
        Simon Willnauer added a comment -

        here is a patch that adds asserts and beefs up a test that fails without the patch

        Show
        Simon Willnauer added a comment - here is a patch that adds asserts and beefs up a test that fails without the patch
        Hide
        Michael McCandless added a comment -

        Hmm, besides deleting files too late, are there any other problems caused here?

        E.g., if an app that only flushes via getReader, would this mean a merge never kicks off (merge starvation)?

        Show
        Michael McCandless added a comment - Hmm, besides deleting files too late, are there any other problems caused here? E.g., if an app that only flushes via getReader, would this mean a merge never kicks off (merge starvation)?
        Hide
        ASF subversion and git services added a comment -

        Commit 1539317 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1539317 ]

        LUCENE-5330: add test

        Show
        ASF subversion and git services added a comment - Commit 1539317 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1539317 ] LUCENE-5330 : add test
        Hide
        ASF subversion and git services added a comment -

        Commit 1539318 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1539318 ]

        LUCENE-5330: add test

        Show
        ASF subversion and git services added a comment - Commit 1539318 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1539318 ] LUCENE-5330 : add test
        Hide
        Michael McCandless added a comment -

        E.g., if an app that only flushes via getReader, would this mean a merge never kicks off (merge starvation)?

        OK, we are fine here (I just committed a test): IW.getReader already calls maybeMerge itself, separately.

        Show
        Michael McCandless added a comment - E.g., if an app that only flushes via getReader, would this mean a merge never kicks off (merge starvation)? OK, we are fine here (I just committed a test): IW.getReader already calls maybeMerge itself, separately.
        Hide
        Michael McCandless added a comment -

        +1 for the patch.

        Show
        Michael McCandless added a comment - +1 for the patch.
        Hide
        Simon Willnauer added a comment -

        mike thanks for the test! Indeed it takes care of merging though. The issue manifests pretty quickly if you use RAM directory and exclusively pull NRT readers. It will simply keep the files that should be deleted around until the queue gets purged.

        Show
        Simon Willnauer added a comment - mike thanks for the test! Indeed it takes care of merging though. The issue manifests pretty quickly if you use RAM directory and exclusively pull NRT readers. It will simply keep the files that should be deleted around until the queue gets purged.
        Hide
        ASF subversion and git services added a comment -

        Commit 1539332 from Simon Willnauer in branch 'dev/trunk'
        [ https://svn.apache.org/r1539332 ]

        LUCENE-5330: Fix IndexWriter to process all internal events on getReader / close / rollback

        Show
        ASF subversion and git services added a comment - Commit 1539332 from Simon Willnauer in branch 'dev/trunk' [ https://svn.apache.org/r1539332 ] LUCENE-5330 : Fix IndexWriter to process all internal events on getReader / close / rollback
        Hide
        ASF subversion and git services added a comment -

        Commit 1539335 from Simon Willnauer in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1539335 ]

        LUCENE-5330: Fix IndexWriter to process all internal events on getReader / close / rollback

        Show
        ASF subversion and git services added a comment - Commit 1539335 from Simon Willnauer in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1539335 ] LUCENE-5330 : Fix IndexWriter to process all internal events on getReader / close / rollback

          People

          • Assignee:
            Simon Willnauer
            Reporter:
            Simon Willnauer
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development