Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      It's a little heavy now, relying too much on reflection (RUE.shallowSizeOf)... and we do this up to 128 times per FST (= per indexed field, per segment, for the terms index).

      1. LUCENE-5884.patch
        8 kB
        Michael McCandless
      2. LUCENE-5884.patch
        5 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Simple patch: I just made Outputs.ramBytesUsed abstract and fixed its subclasses to statically compute the "base" bytes used, and then just add in what's different about each output.

        I'd also like to pre-compute the bytes used by the arc cache and save that away, instead of computing it each time ... next iteration.

        Show
        Michael McCandless added a comment - Simple patch: I just made Outputs.ramBytesUsed abstract and fixed its subclasses to statically compute the "base" bytes used, and then just add in what's different about each output. I'd also like to pre-compute the bytes used by the arc cache and save that away, instead of computing it each time ... next iteration.
        Hide
        Michael McCandless added a comment -

        Another iteration, fixing two other Outputs impls I missed the first time, and pre-caching bytes used when filling the cached root arcs ... I think this is ready.

        Show
        Michael McCandless added a comment - Another iteration, fixing two other Outputs impls I missed the first time, and pre-caching bytes used when filling the cached root arcs ... I think this is ready.
        Hide
        Adrien Grand added a comment -

        Thanks for taking care of this, Caching memory usage of the root arcs in FST.java will certainly help!

        There is just one thing that looked wrong to me: in UpToTwoPositiveIntOutputs.java you return BASE_TWO_INTS_NUM_BYTES + RamUsageEstimator.sizeOf(tl.first) + RamUsageEstimator.sizeOf(tl.second); but I think it should be just BASE_TWO_INTS_NUM_BYTES since the two longs that it carries are not wrappers (java.util.Long) but primitive longs.

        Show
        Adrien Grand added a comment - Thanks for taking care of this, Caching memory usage of the root arcs in FST.java will certainly help! There is just one thing that looked wrong to me: in UpToTwoPositiveIntOutputs.java you return BASE_TWO_INTS_NUM_BYTES + RamUsageEstimator.sizeOf(tl.first) + RamUsageEstimator.sizeOf(tl.second); but I think it should be just BASE_TWO_INTS_NUM_BYTES since the two longs that it carries are not wrappers (java.util.Long) but primitive longs.
        Hide
        Michael McCandless added a comment -

        There is just one thing that looked wrong to me:

        Aha, nice catch! I'll fix that and commit ... thanks.

        Show
        Michael McCandless added a comment - There is just one thing that looked wrong to me: Aha, nice catch! I'll fix that and commit ... thanks.
        Hide
        ASF subversion and git services added a comment -

        Commit 1617940 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1617940 ]

        LUCENE-5884: optimize FST.ramBytesUsed

        Show
        ASF subversion and git services added a comment - Commit 1617940 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1617940 ] LUCENE-5884 : optimize FST.ramBytesUsed
        Hide
        ASF subversion and git services added a comment -

        Commit 1617950 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1617950 ]

        LUCENE-5884: optimize FST.ramBytesUsed

        Show
        ASF subversion and git services added a comment - Commit 1617950 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1617950 ] LUCENE-5884 : optimize FST.ramBytesUsed

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development