[LUCENE-7708] Track PositionLengthAttribute abuse

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.5, 7.0
• Component/s: core/queryparser, modules/analysis
• Assignee: Unassigned
• Reporter: Jim Ferenczi
• Created: 24/Feb/17 11:43
• Updated: 28/Aug/22 15:11
• Resolved: 16/Mar/17 17:45

Description

Some token filters uses the position length attribute of the token stream to encode the number of terms they put in a single token.
This breaks the query parsing because it creates disconnected graph.
I've tracked down the abusive case to 2 candidates:

• ShingleFilter which sets the position length attribute to the length of the shingle.
• CJKBigramFilter which always sets the position length attribute to 2.

I don't think these filters should set the position length at all so the best would be to remove the attribute from these token filters but this could break BWC.
Though this is a serious bug since shingles and cjk bigram now produce invalid queries.