Solr
  1. Solr
  2. SOLR-4074

We should default ramBufferSizeMB to around 100

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      You get diminishing returns after 100, but gains up to 100. In this day and age of RAM, it seems better to default to 100 and have essentially the top indexing speed the buffer can give us.

        Activity

        Hide
        Yonik Seeley added a comment -

        Seems fine. In the past you could get default max heap sizes of 64M in some configurations, but those days are long gone I believe.

        Show
        Yonik Seeley added a comment - Seems fine. In the past you could get default max heap sizes of 64M in some configurations, but those days are long gone I believe.
        Hide
        Simon Willnauer added a comment -

        hey guys, can we do that in lucene too? I mean 32 is a joke IMO...

        Show
        Simon Willnauer added a comment - hey guys, can we do that in lucene too? I mean 32 is a joke IMO...
        Hide
        Mark Miller added a comment -

        Yeah, I actually thought about making the change in lucene too - was tired this morning and don't remember why I decided against. I guess part of it was thinking that Solr is a search server and so it was more important that it default to 100 here. Lucene being a lib, it's was slightly more plausible that lucene could stick with 32 - I didn't remember the reasons it was first set to that.

        I'm +1 on changing in Lucene as well though. I think it it will benefit way more people that it will harm (whom can just lower it). I'll create an issue.

        Show
        Mark Miller added a comment - Yeah, I actually thought about making the change in lucene too - was tired this morning and don't remember why I decided against. I guess part of it was thinking that Solr is a search server and so it was more important that it default to 100 here. Lucene being a lib, it's was slightly more plausible that lucene could stick with 32 - I didn't remember the reasons it was first set to that. I'm +1 on changing in Lucene as well though. I think it it will benefit way more people that it will harm (whom can just lower it). I'll create an issue.
        Hide
        Yonik Seeley added a comment -

        Prob doesn't matter for Lucene - we can set it to whatever, but app developers should always explicitly set it based on their application (i.e. is this a desktop search product? something to run on a tablet?)

        Show
        Yonik Seeley added a comment - Prob doesn't matter for Lucene - we can set it to whatever, but app developers should always explicitly set it based on their application (i.e. is this a desktop search product? something to run on a tablet?)
        Hide
        Mark Miller added a comment -

        but app developers should always explicitly set it based on their application

        My guess is that a large enough group of people take the defaults that it matters. App developers never do what they should

        Show
        Mark Miller added a comment - but app developers should always explicitly set it based on their application My guess is that a large enough group of people take the defaults that it matters. App developers never do what they should
        Hide
        Adrien Grand added a comment -

        hey guys, can we do that in lucene too? I mean 32 is a joke IMO...

        In order Lucene to keep working with small heaps, maybe we should do something similar to the default Java heap size and set ramBufferSizeMB=max(100, maxHeap / N)?

        Show
        Adrien Grand added a comment - hey guys, can we do that in lucene too? I mean 32 is a joke IMO... In order Lucene to keep working with small heaps, maybe we should do something similar to the default Java heap size and set ramBufferSizeMB=max(100, maxHeap / N)?
        Hide
        Yonik Seeley added a comment -

        In order Lucene to keep working with small heaps, maybe we should do something similar to the default Java heap size and set ramBufferSizeMB=max(100, maxHeap / N)?

        Yeah, that sounds like a good idea. N==8?

        Show
        Yonik Seeley added a comment - In order Lucene to keep working with small heaps, maybe we should do something similar to the default Java heap size and set ramBufferSizeMB=max(100, maxHeap / N)? Yeah, that sounds like a good idea. N==8?
        Hide
        Michael McCandless added a comment -

        I don't think we should increase Lucene's default? I think apps that care greatly about indexing performance can tune it upwards ...

        Show
        Michael McCandless added a comment - I don't think we should increase Lucene's default? I think apps that care greatly about indexing performance can tune it upwards ...
        Hide
        Mark Miller added a comment -

        I think a few years ago that may have made sense - but at this point, being anal about 70MB of RAM considering the performance gain seems a little dated. I bet 8/10 new users get better performance from Lucene and .1/10 would find they have to lower it. Probably a sim number as those that already have to lower - I'm not sure there is anything special about 32 if your already fighting for every megabyte.

        Show
        Mark Miller added a comment - I think a few years ago that may have made sense - but at this point, being anal about 70MB of RAM considering the performance gain seems a little dated. I bet 8/10 new users get better performance from Lucene and .1/10 would find they have to lower it. Probably a sim number as those that already have to lower - I'm not sure there is anything special about 32 if your already fighting for every megabyte.
        Hide
        Yonik Seeley added a comment -

        I just committed the change to 100 for Solr to ensure it makes it for 4.1
        If Lucene goes to 100 or something smarter like min(100,maxHeapSize/N), we could change the Solr default to -1 to just pick it up.

        Show
        Yonik Seeley added a comment - I just committed the change to 100 for Solr to ensure it makes it for 4.1 If Lucene goes to 100 or something smarter like min(100,maxHeapSize/N), we could change the Solr default to -1 to just pick it up.
        Hide
        Commit Tag Bot added a comment -

        [trunk commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1415873

        SOLR-4074: Raise default ramBufferSizeMB to 100 from 32.

        Show
        Commit Tag Bot added a comment - [trunk commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1415873 SOLR-4074 : Raise default ramBufferSizeMB to 100 from 32.
        Hide
        Mark Miller added a comment -

        I've also changed the example config so that the commented out default setting matches and I've added a changes entry.

        Show
        Mark Miller added a comment - I've also changed the example config so that the commented out default setting matches and I've added a changes entry.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1415874

        SOLR-4074: Raise default ramBufferSizeMB to 100 from 32.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1415874 SOLR-4074 : Raise default ramBufferSizeMB to 100 from 32.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1415874

        SOLR-4074: Raise default ramBufferSizeMB to 100 from 32.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1415874 SOLR-4074 : Raise default ramBufferSizeMB to 100 from 32.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Yonik Seeley
        http://svn.apache.org/viewvc?view=revision&revision=1410598

        SOLR-4074: default ramBufferSizeMB to 100

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Yonik Seeley http://svn.apache.org/viewvc?view=revision&revision=1410598 SOLR-4074 : default ramBufferSizeMB to 100

          People

          • Assignee:
            Mark Miller
            Reporter:
            Mark Miller
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development