[HIVEMALL-248] UDFs for Kuromoji stoptags

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.5.2
• Fix Version/s: 0.6.0
• Assignee: Makoto Yui
• Reporter: Makoto Yui
• Created: 11/Apr/19 18:15
• Updated: 28/Nov/19 17:53
• Resolved: 13/Apr/19 20:10

Description

In tokenize_ja, user need to provide stoptags that matched tokens removed from the token stream. So, stoptag is "exclusive" rule.

So, create a UDF for "inclusive" rule. stoptags_ja_exclude('名詞') returns array of tags excluding '名詞'. 

https://github.com/apache/lucene-solr/blob/master/lucene/analysis/kuromoji/src/resources/org/apache/lucene/analysis/ja/stoptags.txt

Issue Links

• links to: GitHub Pull Request #189