Lucene - Core
  1. Lucene - Core
  2. LUCENE-3953

Using many fields in filters, sorts causes OOM

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: None
    • Component/s: core/search
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      I have an index with ~1M documents, and many uniquely named fields. Each document only has a few fields populated, but there is a large set of dynamic fields across all docs. In other words, theres many sparse fields.

      Eventually the jvm will fall over due to OOM errors, because lucene's fieldCache grows without bounds. Adding more heap does not help.

        Issue Links

          Activity

          Hide
          Des Lownds added a comment -

          Patch for lucene-solr core 4.0

          Show
          Des Lownds added a comment - Patch for lucene-solr core 4.0
          Hide
          Des Lownds added a comment -

          I would appreciate feedback on the attached patch, which uses a size limited LinkedHashMap for innercache if the system property "lucene.fieldCache.maxFields" is set. If the property is not set, system uses old behavior.

          Show
          Des Lownds added a comment - I would appreciate feedback on the attached patch, which uses a size limited LinkedHashMap for innercache if the system property "lucene.fieldCache.maxFields" is set. If the property is not set, system uses old behavior.
          Hide
          Des Lownds added a comment -

          imports fixed. Patch applies to lucene-solr 4.0 checked out from http://svn.apache.org/repos/asf/lucene/dev/trunk

          Show
          Des Lownds added a comment - imports fixed. Patch applies to lucene-solr 4.0 checked out from http://svn.apache.org/repos/asf/lucene/dev/trunk

            People

            • Assignee:
              Unassigned
              Reporter:
              Des Lownds
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development