Lucene - Core
  1. Lucene - Core
  2. LUCENE-5462

Lucene 3.x producers shouldn't use RamUsageEstimator to estimate size

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      We have had a few Elasticsearch users who have high CPU usage because of RamUsageEstimator being used to estimate memory usage of Lucene3xFields.

        Issue Links

          Activity

          Hide
          Simon Willnauer added a comment -

          damned! We should remove this method or make it a test-only method! At least add it to forbidden APIs we should respin 4.7 for this

          Show
          Simon Willnauer added a comment - damned! We should remove this method or make it a test-only method! At least add it to forbidden APIs we should respin 4.7 for this
          Hide
          Robert Muir added a comment -

          +1, i'm the culprit here. the whole reason for the size estimation method, was that just calling RUE on segmentcorereaders or whatever in my tests was wayy too slow.

          In general there might be more issues than Lucene3xFields. For older deprecated codecs, I "cheated" and used this method, instead of trying to properly implement it... because they were just for back compat. But I can see how an app would use the method, yet still have some old segments...

          Show
          Robert Muir added a comment - +1, i'm the culprit here. the whole reason for the size estimation method, was that just calling RUE on segmentcorereaders or whatever in my tests was wayy too slow. In general there might be more issues than Lucene3xFields. For older deprecated codecs, I "cheated" and used this method, instead of trying to properly implement it... because they were just for back compat. But I can see how an app would use the method, yet still have some old segments...
          Hide
          Simon Willnauer added a comment -

          not your fault but lets make sure this doesn't happen again!

          Show
          Simon Willnauer added a comment - not your fault but lets make sure this doesn't happen again!
          Hide
          Adrien Grand added a comment -

          Here is a patch that removes calls to RamUsageEstimator.sizeOf(Object) in Lucene 3.x terms, norms, term vectors and stored fields. So I think that there is no format producer anymore that relies on this method.

          Show
          Adrien Grand added a comment - Here is a patch that removes calls to RamUsageEstimator.sizeOf(Object) in Lucene 3.x terms, norms, term vectors and stored fields. So I think that there is no format producer anymore that relies on this method.
          Hide
          Simon Willnauer added a comment -

          LGTM

          Show
          Simon Willnauer added a comment - LGTM
          Hide
          Robert Muir added a comment -

          Looks fine!

          Show
          Robert Muir added a comment - Looks fine!
          Hide
          ASF subversion and git services added a comment -

          Commit 1570240 from Adrien Grand in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1570240 ]

          LUCENE-5462: Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

          Show
          ASF subversion and git services added a comment - Commit 1570240 from Adrien Grand in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1570240 ] LUCENE-5462 : Don't use RamUsageEstimator.sizeOf(Object) in codec producers.
          Hide
          ASF subversion and git services added a comment -

          Commit 1570243 from Adrien Grand in branch 'dev/branches/lucene_solr_4_7'
          [ https://svn.apache.org/r1570243 ]

          LUCENE-5462: Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

          Show
          ASF subversion and git services added a comment - Commit 1570243 from Adrien Grand in branch 'dev/branches/lucene_solr_4_7' [ https://svn.apache.org/r1570243 ] LUCENE-5462 : Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

            People

            • Assignee:
              Adrien Grand
              Reporter:
              Adrien Grand
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development