[LUCENE-2522] add simple japanese tokenizer, based on tinysegmenter

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Fix Version/s: 4.9, 6.0
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 01/Jul/10 16:47
• Updated: 28/Aug/22 12:28

Description

TinySegmenter (http://www.chasen.org/~taku/software/TinySegmenter/) is a tiny japanese segmenter.

It was ported to java/lucene by Kohei TAKETA <[e-mail]>,
and is under friendly license terms (BSD, some files explicitly disclaim copyright to the source code, giving a blessing instead)

Koji knows the author, and already contacted about incorporating into lucene:

I've contacted Takeda-san who is the creater of Java version of
TinySegmenter. He said he is happy if his program is part of Lucene.
He is a co-author of my book about Solr published in Japan, BTW. ;-)