[SOLR-1423] Lucene 2.9 RC4 may need some changes in Solr Analyzers using CharStream & others

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4
• Fix Version/s: 1.4
• Component/s: Schema and Analysis
• Assignee: Koji Sekiguchi
• Reporter: Uwe Schindler
• Created: 11/Sep/09 12:08
• Updated: 10/Nov/09 15:52
• Resolved: 18/Sep/09 07:38

Description

Because of some backwards compatibility problems (LUCENE-1906) we changed the CharStream/CharFilter API a little bit. Tokenizer now only has a input field of type java.io.Reader (as before the CharStream code). To correct offsets, it is now needed to call the Tokenizer.correctOffset(int) method, which delegates to the CharStream (if input is subclass of CharStream), else returns an uncorrected offset. Normally it is enough to change all occurences of input.correctOffset() to this.correctOffset() in Tokenizers. It should also be checked, if custom Tokenizers in Solr do correct their offsets.

Issue Links

• relates to: LUCENE-1906 Backwards problems with CharStream and Tokenizers with custom reset(Reader) method
• relates to: SOLR-1404 Random failures with highlighting