[LUCENE-1227] NGramTokenizer to handle more than 1024 chars

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Unassigned
• Reporter: Hiroaki Kawai
• Created: 13/Mar/08 08:22
• Updated: 28/Aug/22 11:48
• Resolved: 26/Apr/13 22:11

Description

Current NGramTokenizer can't handle character stream that is longer than 1024. This is too short for non-whitespace-separated languages.

I created a patch for this issues.

Issue Links

• incorporates: LUCENE-1225 NGramTokenizer creates bad TokenStream