[LUCENE-7468] ASCIIFoldingFilter should not emit duplicated tokens when preserve original is on

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk, 4.7
• Fix Version/s: 6.3, 7.0
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Unassigned
• Reporter: David Causse
• Created: 28/Sep/16 13:13
• Updated: 28/Aug/22 15:03
• Resolved: 06/Oct/16 08:58

Description

The ASCIIFoldingFilter seems to make the bold assumption that any tokens that contain a char outside the ASCII range will be folded.
The problem is that when preserve original is true we capture and restore the state even if the token remains unmodified.
This causes term frequencies to double for such words and probably extra space used when positions/offsets are stored in the postings.