Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      IndexWriter uses AppendingLongBuffer in several places, and in a few of them the mapping is monotonically increasing so we could save additional space by only encoding the delta from a linear projection.

      1. LUCENE-4780.patch
        23 kB
        Adrien Grand

        Activity

        Hide
        Adrien Grand added a comment -

        Patch. I tried to replace AppendingLongBuffer with MonotonicAppendingLongBuffer wherever it makes sense.

        Show
        Adrien Grand added a comment - Patch. I tried to replace AppendingLongBuffer with MonotonicAppendingLongBuffer wherever it makes sense.
        Hide
        Robert Muir added a comment -

        So this works even if its "mostly monotonic" but not always?

        In LUCENE-4765, i refactored SortedBytes merging to use OrdinalMap, so if this works we can
        also reduce RAM used during merging, which would be awesome. But using it in merge means that when there are
        "deleted terms" its ordinal mappings are not always strictly monotonic, but probably mostly so.

        In this case we would just get some negative deviations right?

        Show
        Robert Muir added a comment - So this works even if its "mostly monotonic" but not always? In LUCENE-4765 , i refactored SortedBytes merging to use OrdinalMap, so if this works we can also reduce RAM used during merging, which would be awesome. But using it in merge means that when there are "deleted terms" its ordinal mappings are not always strictly monotonic, but probably mostly so. In this case we would just get some negative deviations right?
        Hide
        Adrien Grand added a comment -

        So this works even if its "mostly monotonic" but not always?

        Even if not at all. The test case even tests it with completely random longs.

        In this case we would just get some negative deviations right?

        Right.

        Show
        Adrien Grand added a comment - So this works even if its "mostly monotonic" but not always? Even if not at all. The test case even tests it with completely random longs. In this case we would just get some negative deviations right? Right.
        Hide
        Adrien Grand added a comment -

        Committed on the LUCENE-4765 branch.

        Show
        Adrien Grand added a comment - Committed on the LUCENE-4765 branch.
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Adrien Grand
            Reporter:
            Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development