[LUCENE-9283] DelimitedBoostTokenFilter can fail testRandomChains

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: master (9.0), 8.6
• Lucene Fields: New
• Assignee: Alan Woodward
• Reporter: Alan Woodward
• Created: 18/Mar/20 09:33
• Updated: 15/Jul/20 15:10
• Resolved: 19/Mar/20 13:47

Description

DelimitedBoostTokenFilter expects tokens of the form `token` or `token|number` and throws a NumberFormatException if the `number` part can't be parsed. This can cause test failures when we build random chains and throw random data through them.

We can either exclude DelimiteBoostTokenFilter when building a random analyzer, or add a flag to ignore badly-formed tokens. I lean towards doing the former, as I don't really want to make leniency the default here.