[SOLR-11139] FreeTextSuggester exits on first error while building suggester index

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.5.4
• Component/s: Suggester
• Assignee: Unassigned
• Reporter: Angel Todorov
• Created: 24/Jul/17 19:54
• Updated: 08/Jun/19 15:23

Description

If an exception is thrown below, it will exit the while loop, therefore stopping building of the suggester index. In fact, it will not build anything, because the exception is not handled in any way. This is highly unacceptable, because in an enterprise environment there will always be data that does not conform to the expected rules.

while (true) {
BytesRef term = termsEnum.next();
if (term == null)

{ break; }

int ngramCount = countGrams(term);
if (ngramCount > grams)

{ throw new IllegalArgumentException("tokens must not contain separator byte; got token=" + term + " but gramCount=" + ngramCount + ", which is greater than expected max ngram size=" + grams); }

if (ngramCount == 1)

{ totTokens += termsEnum.totalTermFreq(); }

builder.add(Util.toIntsRef(term, scratchInts), encodeWeight(termsEnum.totalTermFreq()));
}