Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5462

Lucene 3.x producers shouldn't use RamUsageEstimator to estimate size

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 4.7
        • Component/s: None
        • Labels:
          None
        • Lucene Fields:
          New

          Description

          We have had a few Elasticsearch users who have high CPU usage because of RamUsageEstimator being used to estimate memory usage of Lucene3xFields.

            Issue Links

            is related to

            Bug - A problem which impairs or prevents the functions of the product. LUCENE-5940 change index backwards compatibility policy.

            • Major - Major loss of function.
            • Open

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              simonw Simon Willnauer added a comment -

              damned! We should remove this method or make it a test-only method! At least add it to forbidden APIs we should respin 4.7 for this

              Show
              simonw Simon Willnauer added a comment - damned! We should remove this method or make it a test-only method! At least add it to forbidden APIs we should respin 4.7 for this
              Hide
              Permalink
              rcmuir Robert Muir added a comment -

              +1, i'm the culprit here. the whole reason for the size estimation method, was that just calling RUE on segmentcorereaders or whatever in my tests was wayy too slow.

              In general there might be more issues than Lucene3xFields. For older deprecated codecs, I "cheated" and used this method, instead of trying to properly implement it... because they were just for back compat. But I can see how an app would use the method, yet still have some old segments...

              Show
              rcmuir Robert Muir added a comment - +1, i'm the culprit here. the whole reason for the size estimation method, was that just calling RUE on segmentcorereaders or whatever in my tests was wayy too slow. In general there might be more issues than Lucene3xFields. For older deprecated codecs, I "cheated" and used this method, instead of trying to properly implement it... because they were just for back compat. But I can see how an app would use the method, yet still have some old segments...
              Hide
              Permalink
              simonw Simon Willnauer added a comment -

              not your fault but lets make sure this doesn't happen again!

              Show
              simonw Simon Willnauer added a comment - not your fault but lets make sure this doesn't happen again!
              Hide
              Permalink
              jpountz Adrien Grand added a comment -

              Here is a patch that removes calls to RamUsageEstimator.sizeOf(Object) in Lucene 3.x terms, norms, term vectors and stored fields. So I think that there is no format producer anymore that relies on this method.

              Show
              jpountz Adrien Grand added a comment - Here is a patch that removes calls to RamUsageEstimator.sizeOf(Object) in Lucene 3.x terms, norms, term vectors and stored fields. So I think that there is no format producer anymore that relies on this method.
              Hide
              Permalink
              simonw Simon Willnauer added a comment -

              LGTM

              Show
              simonw Simon Willnauer added a comment - LGTM
              Hide
              Permalink
              rcmuir Robert Muir added a comment -

              Looks fine!

              Show
              rcmuir Robert Muir added a comment - Looks fine!
              Hide
              Permalink
              jira-bot ASF subversion and git services added a comment -

              Commit 1570240 from Adrien Grand in branch 'dev/branches/branch_4x'
              [ https://svn.apache.org/r1570240 ]

              LUCENE-5462: Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

              Show
              jira-bot ASF subversion and git services added a comment - Commit 1570240 from Adrien Grand in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1570240 ] LUCENE-5462 : Don't use RamUsageEstimator.sizeOf(Object) in codec producers.
              Hide
              Permalink
              jira-bot ASF subversion and git services added a comment -

              Commit 1570243 from Adrien Grand in branch 'dev/branches/lucene_solr_4_7'
              [ https://svn.apache.org/r1570243 ]

              LUCENE-5462: Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

              Show
              jira-bot ASF subversion and git services added a comment - Commit 1570243 from Adrien Grand in branch 'dev/branches/lucene_solr_4_7' [ https://svn.apache.org/r1570243 ] LUCENE-5462 : Don't use RamUsageEstimator.sizeOf(Object) in codec producers.

                People

                • Assignee:
                  jpountz Adrien Grand
                  Reporter:
                  jpountz Adrien Grand
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development