Lucene - Core
  1. Lucene - Core
  2. LUCENE-5254

SegmentCoreReader's "owner" reference back to the first SegmentReader causes leaks

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.6, 6.0
    • Fix Version/s: 4.5.1, 4.6, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Spinoff from LUCENE-5248, where Shai discovered this ...

      SegmentCoreReaders has a SegmentReader owner member, that points to the
      first SegmentReader that was opened. When that SR is reopened to SR2,
      e.g. because new deletes or NDV updates happened, the same SCR is
      shared.

      But, even if you close SR1, any thing it points to cannot be GCd
      because SCR is pointing to it.

      I think the big things are liveDocs and the NDV update maps; Shai is
      going to fix the latter in LUCENE-5248, so this issue should fix
      liveDocs.

      The simplest fix is to make liveDocs not final and null it out in
      doClose ... but that's sort of fragile (what if we add other members
      in the future and forget to null them on close?). I think it'd be better to
      eliminate the owner reference; it's only used so we can evict
      FieldCache entry once the core is closed. Maybe we can just store the
      coreCacheKey instead?

      1. LUCENE-5254.patch
        12 kB
        Michael McCandless

        Activity

        Hide
        Simon Willnauer added a comment -

        +1 to use the core key only. I checked what ES does with it and we always pull the core key here anyways.

        Show
        Simon Willnauer added a comment - +1 to use the core key only. I checked what ES does with it and we always pull the core key here anyways.
        Hide
        Uwe Schindler added a comment -

        +1 To nuke this reference and only use the core cache key! Please dont make things in SegmentReaders unfinal! This is bad design!

        It took some time until Shai and I understood why RAMUsageEstimator showed so much RAM. But this was a real good use case for the Estimator. Because it showd the bug

        Show
        Uwe Schindler added a comment - +1 To nuke this reference and only use the core cache key! Please dont make things in SegmentReaders unfinal! This is bad design! It took some time until Shai and I understood why RAMUsageEstimator showed so much RAM. But this was a real good use case for the Estimator. Because it showd the bug
        Hide
        Michael McCandless added a comment -

        Patch, changing SCR to hold the coreCacheKey, not the SegmentReader, and changing FC.purge(AtomicReader) -> FC.purgeByCacheKey(Object).

        Show
        Michael McCandless added a comment - Patch, changing SCR to hold the coreCacheKey, not the SegmentReader, and changing FC.purge(AtomicReader) -> FC.purgeByCacheKey(Object).
        Hide
        ASF subversion and git services added a comment -

        Commit 1529135 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1529135 ]

        LUCENE-5254: don't hold ref to original SR from SCR, to avoid bounded leak of things like live docs bitset

        Show
        ASF subversion and git services added a comment - Commit 1529135 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1529135 ] LUCENE-5254 : don't hold ref to original SR from SCR, to avoid bounded leak of things like live docs bitset
        Hide
        ASF subversion and git services added a comment -

        Commit 1529136 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1529136 ]

        LUCENE-5254: don't hold ref to original SR from SCR, to avoid bounded leak of things like live docs bitset

        Show
        ASF subversion and git services added a comment - Commit 1529136 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1529136 ] LUCENE-5254 : don't hold ref to original SR from SCR, to avoid bounded leak of things like live docs bitset
        Hide
        ASF subversion and git services added a comment -

        Commit 1529139 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1529139 ]

        LUCENE-5254: just pass 'this' to the CoreClosedListeners

        Show
        ASF subversion and git services added a comment - Commit 1529139 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1529139 ] LUCENE-5254 : just pass 'this' to the CoreClosedListeners
        Hide
        ASF subversion and git services added a comment -

        Commit 1529141 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1529141 ]

        LUCENE-5254: just pass 'this' to the CoreClosedListeners

        Show
        ASF subversion and git services added a comment - Commit 1529141 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1529141 ] LUCENE-5254 : just pass 'this' to the CoreClosedListeners
        Hide
        ASF subversion and git services added a comment -

        Commit 1530845 from Robert Muir in branch 'dev/branches/lucene_solr_4_5'
        [ https://svn.apache.org/r1530845 ]

        LUCENE-4998, LUCENE-5242, LUCENE-5254, LUCENE-5262, LUCENE-5263, LUCENE-5264: svn merge -c 1522723 -c 1525896 -c 1529136 -c 1529141 -c 1530063 -c 1530416 -c 1530657

        Show
        ASF subversion and git services added a comment - Commit 1530845 from Robert Muir in branch 'dev/branches/lucene_solr_4_5' [ https://svn.apache.org/r1530845 ] LUCENE-4998 , LUCENE-5242 , LUCENE-5254 , LUCENE-5262 , LUCENE-5263 , LUCENE-5264 : svn merge -c 1522723 -c 1525896 -c 1529136 -c 1529141 -c 1530063 -c 1530416 -c 1530657

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development