[NUTCH-224] Nutch doesn't handle Korean text at all

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.7.1
• Component/s: indexer
• Assignee: Unassigned
• Reporter: Kuro Kurosaka
• Created: 07/Mar/06 06:56
• Updated: 01/Apr/11 14:58
• Resolved: 01/Apr/11 14:58

Description

I was browing NutchAnalysis.jj and found that
Hungul Syllables (U+AC00 ... U+D7AF; U+xxxx means
a Unicode character of the hex value xxxx) are not
part of LETTER or CJK class. This seems to me that
Nutch cannot handle Korean documents at all.

I posted the above message at nutch-user ML and Cheolgoo Kang [[e-mail]]
replied as:
------------------------------------------------------------------------------------
There was similar issue with Lucene's StandardTokenizer.jj.

http://issues.apache.org/jira/browse/LUCENE-444

and

http://issues.apache.org/jira/browse/LUCENE-461

I'm have almost no experience with Nutch, but you can handle it like
those issues above.
------------------------------------------------------------------------------------

Both fixes should probably be ported back to NuatchAnalysis.jj.