[LUCENE-3897] KuromojiTokenizer fails with large docs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Christian Moen
• Reporter: Robert Muir
• Created: 21/Mar/12 02:49
• Updated: 28/Aug/22 13:11
• Resolved: 22/Mar/12 12:18

Description

just shoving largeish random docs triggers asserts like:

    [junit] Caused by: java.lang.AssertionError: backPos=4100 vs lastBackTracePos=5120
    [junit] 	at org.apache.lucene.analysis.kuromoji.KuromojiTokenizer.backtrace(KuromojiTokenizer.java:907)
    [junit] 	at org.apache.lucene.analysis.kuromoji.KuromojiTokenizer.parse(KuromojiTokenizer.java:756)
    [junit] 	at org.apache.lucene.analysis.kuromoji.KuromojiTokenizer.incrementToken(KuromojiTokenizer.java:403)
    [junit] 	at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:404)

But, you get no seed...

I'll commit the test case and @Ignore it.