[SOLR-2597] XmlCharFilter

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0-ALPHA
• Component/s: Schema and Analysis
• Assignee: Unassigned
• Reporter: Michael Sokolov
• Created: 15/Jun/11 13:10
• Updated: 17/Jun/11 02:42

Description

This CharFilter processes incoming XML using the Woodstox parser, stripping all non-text content and remembering offsets, just like HTMLCharFilter, but respecting XML conventions like XML entities defined in a DTD. XmlCharFilter also provides the ability to exclude (and include) the content of certain named elements.

In order to compute character offsets properly when mixed line termination styles are present (\r, \r\n), or when XML character entities (<, ", &) are present, we require a newer version of Woodstox (4.1.1) than is currently in solr/lib. The earlier versions of the parser could not report these entity events, so we couldn't tell the difference between "<" and "<" and the offsets could be wrong. The upgraded version is in the patch.