Lucene - Core
  1. Lucene - Core
  2. LUCENE-6501

Flatten subreader structure in ParallelCompositeReader

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.2
    • Fix Version/s: 5.3, 6.0
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The current implementation of ParallelCompositeReader reassembles the whole subreader structure of the wrapped reader with ParallelLeafReader and ParallelCompositeReader.

      This leads to bugs like described in LUCENE-6500. This reaches back to the time when this reader was reimplemented for the first time shortly before release of 4.0. Shortly afterwards, we completely changed our search infrastructure to just call leaves() and working with them. The method getSequentialSubReaders was made protected, just to be implemented by subclasses (like this one). But no external code can ever call it. Also the search API just rely on the baseId in relation to the top-level reader (to correctly present document ids). The structure is completely unimportant.

      This issue will therefore simplify ParallelCompositeReader to just fetch all LeafReaders and build a flat structure of ParallelLeafReaders from it. This also has the nice side-effect, that only the parallel leaf readers must be equally sized, not their structure.

      This issue will solve LUCENE-6500 as a side effect. I just opened a new issue for discussion and to have this listed as "feature" and not bug.

      In general, we could also hide the ParallelLeafReader class and make it an implementation detail. ParallelCompositeReader would be the only entry point -> because people could pass any IndexReader structure in, a single AtomicReader would just produce a CompositeReader with one leaf. We could then also rename it back to ParallelReader (like it was in pre Lucene4).

      1. LUCENE-6501.patch
        13 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment -

          Patch.

          Show
          Uwe Schindler added a comment - Patch.
          Hide
          Uwe Schindler added a comment -

          I am currently testing Lucene Core in a loop to ensure that tests pass even with sometimes flattened down (we may add an extra reader wrapper for this in test-framework FlattenedCompositeReader).

          I am confident that all is fine, because nothing problematic calls getSetSequentialSubReaders() or IndexReaderContext.children() or IndexReaderContext.parent

          Show
          Uwe Schindler added a comment - I am currently testing Lucene Core in a loop to ensure that tests pass even with sometimes flattened down (we may add an extra reader wrapper for this in test-framework FlattenedCompositeReader). I am confident that all is fine, because nothing problematic calls getSetSequentialSubReaders() or IndexReaderContext.children() or IndexReaderContext.parent
          Hide
          Adrien Grand added a comment -

          +1

          Show
          Adrien Grand added a comment - +1
          Hide
          Uwe Schindler added a comment -

          No problems with testing up to now. I think this is safe for 5.3 (not 5.2) so it can bake longer.

          Show
          Uwe Schindler added a comment - No problems with testing up to now. I think this is safe for 5.3 (not 5.2) so it can bake longer.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681998 from Uwe Schindler in branch 'dev/trunk'
          [ https://svn.apache.org/r1681998 ]

          LUCENE-6501: Flatten subreader structure in ParallelCompositeReader (fixes close listener bug LUCENE-6500)

          Show
          ASF subversion and git services added a comment - Commit 1681998 from Uwe Schindler in branch 'dev/trunk' [ https://svn.apache.org/r1681998 ] LUCENE-6501 : Flatten subreader structure in ParallelCompositeReader (fixes close listener bug LUCENE-6500 )
          Hide
          ASF subversion and git services added a comment -

          Commit 1682000 from Uwe Schindler in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1682000 ]

          Merged revision(s) 1681998 from lucene/dev/trunk:
          LUCENE-6501: Flatten subreader structure in ParallelCompositeReader (fixes close listener bug LUCENE-6500)

          Show
          ASF subversion and git services added a comment - Commit 1682000 from Uwe Schindler in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1682000 ] Merged revision(s) 1681998 from lucene/dev/trunk: LUCENE-6501 : Flatten subreader structure in ParallelCompositeReader (fixes close listener bug LUCENE-6500 )
          Hide
          Shalin Shekhar Mangar added a comment -

          Bulk close for 5.3.0 release

          Show
          Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development