[LUCY-191] Unicode normalization

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.3.0 (incubating)
• Component/s: Analysis
• Labels: patch
• Assignee: Nikolas Wellnhofer
• Reporter: Nikolas Wellnhofer
• Created: 19/Nov/11 17:04
• Updated: 13/Dec/11 00:41
• Resolved: 13/Dec/11 00:41

Description

As discussed on the mailing list, it would be nice to have Unicode normalization, Unicode case folding and stripping of accents as part of the analyzer chain. With the help of utf8proc this can be done in one pass. So I proposed a new analyzer Lucy::Analyzer::Normalizer with an interface described here:

http://mail-archives.apache.org/mod_mbox/incubator-lucy-dev/201111.mbox/%3C4EC43816.1070107%40aevum.de%3E