Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2026

Refactoring of IndexWriter



    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: core/index
    • Labels:
    • Lucene Fields:


      I've been thinking for a while about refactoring the IndexWriter into
      two main components.

      One could be called a SegmentWriter and as the
      name says its job would be to write one particular index segment. The
      default one just as today will provide methods to add documents and
      flushes when its buffer is full.
      Other SegmentWriter implementations would do things like e.g. appending or
      copying external segments [what addIndexes*() currently does].

      The second component's job would it be to manage writing the segments
      file and merging/deleting segments. It would know about
      DeletionPolicy, MergePolicy and MergeScheduler. Ideally it would
      provide hooks that allow users to manage external data structures and
      keep them in sync with Lucene's data during segment merges.

      API wise there are things we have to figure out, such as where the
      updateDocument() method would fit in, because its deletion part
      affects all segments, whereas the new document is only being added to
      the new segment.

      Of course these should be lower level APIs for things like parallel
      indexing and related use cases. That's why we should still provide
      easy to use APIs like today for people who don't need to care about
      per-segment ops during indexing. So the current IndexWriter could
      probably keeps most of its APIs and delegate to the new classes.


          Issue Links



              • Assignee:
                michaelbusch Michael Busch
                michaelbusch Michael Busch
              • Votes:
                1 Vote for this issue
                7 Start watching this issue


                • Created: