Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      Today, the use of addIndexes and addIndexesNoOptimize is confusing -
      especially on when to invoke each. Also, addIndexes calls optimize() in
      the beginning, but only on the target index. It also includes the
      following jdoc statement, which from how I understand the code, is
      wrong: After this completes, the index is optimized. – optimize() is
      called in the beginning and not in the end.

      On the other hand, addIndexesNoOptimize does not call optimize(), and
      relies on the MergeScheduler and MergePolicy to handle the merges.

      After a short discussion about that on the list (Thanks Mike for the
      clarifications!) I understand that there are really two core differences
      between the two:

      • addIndexes supports IndexReader extensions
      • addIndexesNoOptimize performs better

      This issue proposes the following:

      1. Clear up the documentation of each, spelling out the pros/cons of
        calling them clearly in the javadocs.
      2. Rename addIndexesNoOptimize to addIndexes
      3. Remove optimize() call from addIndexes(IndexReader...)
      4. Document that clearly in both, w/ a recommendation to call optimize()
        before on any of the Directories/Indexes if it's a concern.

      That way, we maintain all the flexibility in the API -
      addIndexes(IndexReader...) allows for using IR extensions,
      addIndexes(Directory...) is considered more efficient, by allowing the
      merges to happen concurrently (depending on MS) and also factors in the
      MP. So unless you have an IR extension, addDirectories is really the one
      you should be using. And you have the freedom to call optimize() before
      each if you care about it, or don't if you don't care. Either way,
      incurring the cost of optimize() is entirely in the user's hands.

      BTW, addIndexes(IndexReader...) does not use neither the MergeScheduler
      nor MergePolicy, but rather call SegmentMerger directly. This might be
      another place for improvement. I'll look into it, and if it's not too
      complicated, I may cover it by this issue as well. If you have any hints
      that can give me a good head start on that, please don't be shy .

      1. LUCENE-2455_trunk.patch
        141 kB
        Shai Erera
      2. LUCENE-2455_3x.patch
        56 kB
        Shai Erera
      3. LUCENE-2455_3x.patch
        77 kB
        Shai Erera
      4. LUCENE-2455_3x.patch
        105 kB
        Shai Erera
      5. LUCENE-2455_3x.patch
        105 kB
        Shai Erera
      6. LUCENE-2455_3x.patch
        105 kB
        Shai Erera
      7. index.31.nocfs.zip
        9 kB
        Uwe Schindler
      8. index.31.cfs.zip
        5 kB
        Uwe Schindler

        Activity

        Grant Ingersoll made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12563413 ] jira [ 12584201 ]
        Mark Thomas made changes -
        Workflow jira [ 12510566 ] Default workflow, editable Closed status [ 12563413 ]
        Shai Erera made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Lucene Fields [New] [New, Patch Available]
        Resolution Fixed [ 1 ]
        Uwe Schindler made changes -
        Attachment index.31.cfs.zip [ 12445632 ]
        Attachment index.31.nocfs.zip [ 12445633 ]
        Shai Erera made changes -
        Attachment LUCENE-2455_trunk.patch [ 12445628 ]
        Shai Erera made changes -
        Attachment LUCENE-2455_3x.patch [ 12445469 ]
        Shai Erera made changes -
        Attachment LUCENE-2455_3x.patch [ 12445427 ]
        Shai Erera made changes -
        Attachment LUCENE-2455_3x.patch [ 12445252 ]
        Shai Erera made changes -
        Attachment LUCENE-2455_3x.patch [ 12445232 ]
        Shai Erera made changes -
        Field Original Value New Value
        Attachment LUCENE-2455_3x.patch [ 12444493 ]
        Shai Erera created issue -

          People

          • Assignee:
            Shai Erera
            Reporter:
            Shai Erera
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development