Lucene - Core
  1. Lucene - Core
  2. LUCENE-5462

Lucene 3.x producers shouldn't use RamUsageEstimator to estimate size

        Details

        • Type: Bug Bug
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 4.7
        • Component/s: None
        • Labels:
          None
        • Lucene Fields:
          New

          Description

          We have had a few Elasticsearch users who have high CPU usage because of RamUsageEstimator being used to estimate memory usage of Lucene3xFields.

            Issue Links

            is related to

            Bug - A problem which impairs or prevents the functions of the product. LUCENE-5940 change index backwards compatibility policy.

            • Major - Major loss of function.
            • Open

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Simon Willnauer added a comment -

              damned! We should remove this method or make it a test-only method! At least add it to forbidden APIs we should respin 4.7 for this

              Show
              Simon Willnauer added a comment - damned! We should remove this method or make it a test-only method! At least add it to forbidden APIs we should respin 4.7 for this
              Hide
              Permalink
              Robert Muir added a comment -

              +1, i'm the culprit here. the whole reason for the size estimation method, was that just calling RUE on segmentcorereaders or whatever in my tests was wayy too slow.

              In general there might be more issues than Lucene3xFields. For older deprecated codecs, I "cheated" and used this method, instead of trying to properly implement it... because they were just for back compat. But I can see how an app would use the method, yet still have some old segments...

              Show
              Robert Muir added a comment - +1, i'm the culprit here. the whole reason for the size estimation method, was that just calling RUE on segmentcorereaders or whatever in my tests was wayy too slow. In general there might be more issues than Lucene3xFields. For older deprecated codecs, I "cheated" and used this method, instead of trying to properly implement it... because they were just for back compat. But I can see how an app would use the method, yet still have some old segments...
              Hide
              Permalink
              Simon Willnauer added a comment -

              not your fault but lets make sure this doesn't happen again!

              Show
              Simon Willnauer added a comment - not your fault but lets make sure this doesn't happen again!
              Hide
              Permalink
              Adrien Grand added a comment -

              Here is a patch that removes calls to RamUsageEstimator.sizeOf(Object) in Lucene 3.x terms, norms, term vectors and stored fields. So I think that there is no format producer anymore that relies on this method.

              Show
              Adrien Grand added a comment - Here is a patch that removes calls to RamUsageEstimator.sizeOf(Object) in Lucene 3.x terms, norms, term vectors and stored fields. So I think that there is no format producer anymore that relies on this method.
              Hide
              Permalink
              Simon Willnauer added a comment -

              LGTM

              Show
              Simon Willnauer added a comment - LGTM
              Hide
              Permalink
              Robert Muir added a comment -

              Looks fine!

              Show
              Robert Muir added a comment - Looks fine!
              Hide
              Permalink
              ASF subversion and git services added a comment -

              Commit 1570240 from Adrien Grand in branch 'dev/branches/branch_4x'
              [ https://svn.apache.org/r1570240 ]

              LUCENE-5462: Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

              Show
              ASF subversion and git services added a comment - Commit 1570240 from Adrien Grand in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1570240 ] LUCENE-5462 : Don't use RamUsageEstimator.sizeOf(Object) in codec producers.
              Hide
              Permalink
              ASF subversion and git services added a comment -

              Commit 1570243 from Adrien Grand in branch 'dev/branches/lucene_solr_4_7'
              [ https://svn.apache.org/r1570243 ]

              LUCENE-5462: Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

              Show
              ASF subversion and git services added a comment - Commit 1570243 from Adrien Grand in branch 'dev/branches/lucene_solr_4_7' [ https://svn.apache.org/r1570243 ] LUCENE-5462 : Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

                People

                • Assignee:
                  Adrien Grand
                  Reporter:
                  Adrien Grand
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development