[LUCENE-3026] smartcn analyzer throw NullPointer exception when the length of analysed text over 32767

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1, 4.0-ALPHA
• Fix Version/s: 3.2, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: wangzhenghang
• Created: 13/Apr/11 15:38
• Updated: 22/Nov/24 15:01
• Resolved: 14/Apr/11 15:38

Description

That's all because of org.apache.lucene.analysis.cn.smart.hhmm.SegGraph's makeIndex() method:
public List<SegToken> makeIndex() {
List<SegToken> result = new ArrayList<SegToken>();
int s = -1, count = 0, size = tokenListTable.size();
List<SegToken> tokenList;
short index = 0;
while (count < size) {
if (isStartExist(s)) {
tokenList = tokenListTable.get(s);
for (SegToken st : tokenList)

{ st.index = index; result.add(st); index++; }

count++;
}
s++;
}
return result;
}

here 'short index = 0;' should be 'int index = 0;'. And that's reported here http://code.google.com/p/imdict-chinese-analyzer/issues/detail?id=2 and http://code.google.com/p/imdict-chinese-analyzer/issues/detail?id=11, the author XiaoPingGao have already fixed this bug:http://code.google.com/p/imdict-chinese-analyzer/source/browse/trunk/src/org/apache/lucene/analysis/cn/smart/hhmm/SegGraph.java