[SOLR-10014] Log a warning when the number of fields in a core exceeds a configurable value

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 4.10.4
• Assignee: Unassigned
• Reporter: Shawn Heisey
• Created: 20/Jan/17 20:11
• Updated: 08/Jun/19 15:32

Description

When the number of fields in an index gets extremely large, major performance problems can occur. If the number of fields in a core exceeds a configurable number, with a default somewhere around 10000, a warning should be logged when the SolrCore is first created. A decision needs to be made about whether to repeat the warning on core reload ... my instinct is that it should NOT be repeated, but I can see where a repeat might have some value. Logging on reloads as well as startup would likely be easier.

This was discovered by a Solr user who had a 420MB index with 650K documents, but their applications were abusing dynamic fields to the point where they had about 2 million unique fields in the index. The small size of the index should have resulted in extremely fast commit times, but commits were taking about 10 seconds because of what Lucene had to do to handle all those fields.

Issue Links

• is related to: LUCENE-7648 Millions of fields in an index makes some operations slow, opening a new searcher in particular