[LUCENE-2074] Use a separate JFlex generated Unicode 4 by Java 5 compatible StandardTokenizer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0
• Fix Version/s: 3.1, 4.0-ALPHA
• Lucene Fields: New, Patch Available
• Assignee: Uwe Schindler
• Reporter: Uwe Schindler
• Created: 16/Nov/09 21:30
• Updated: 28/Aug/22 12:14
• Resolved: 08/Apr/10 22:57

Description

The current trunk version of StandardTokenizerImpl was generated by Java 1.4 (according to the warning). In Java 3.0 we switch to Java 1.5, so we should regenerate the file.

After regeneration the Tokenizer behaves different for some characters. Because of that we should only use the new TokenizerImpl when Version.LUCENE_30 or LUCENE_31 is used as matchVersion.

Issue Links

• incorporates: LUCENE-2384 Reset zzBuffer in StandardTokenizerImpl* when lexer is reset.
• is depended upon by: LUCENE-2244 Improve StandardTokenizer's understanding of non ASCII punctuation and quotes

Sub-Tasks

• Reset zzBuffer in StandardTokenizerImpl* when lexer is reset.: Closed