Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7648

Millions of fields in an index makes some operations slow, opening a new searcher in particular

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 4.10.4
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Got a Solr user who was experiencing very slow commit times on their index – 10 seconds or more. This is on a 650K document index sized at about 420MB, with all Solr cache autowarm counts at zero.

      After some profiling of their Solr install, they finally determined that the problem was an abuse of dynamic fields. The largest .fnm file in their index was 130MB, with the total of all .fnm files at 140MB. The user estimates that they have about 2 million fields in this index. They will be fixing the situation so the field count is more reasonable.

      While I do understand that millions of fields in an index is a pathological setup, and that some parts of Lucene operation are always going to be slow on an index like that, 10 seconds for a new searcher seemed excessive to me. Perhaps there is an opportunity for a little bit of optimization?

      The version is old – 4.10.4. They have not yet tried a newer version.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                elyograg Shawn Heisey
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: