[LUCENE-2732] Fix charset problems in XML loading in HyphenationCompoundWordTokenFilter (also Solr's loader from schema)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.4, 3.0.3, 3.1, 4.0-ALPHA
• Lucene Fields: New
• Assignee: Uwe Schindler
• Reporter: Uwe Schindler
• Created: 31/Oct/10 12:12
• Updated: 28/Aug/22 12:35
• Resolved: 31/Oct/10 14:51

Description

As said in LUCENE-2731, the handling of XML in HyphenationCompoundWordTokenFilter is broken and breaks XML 1.0 (5th edition) spec totally. You should never supply a Reader to any XML api, unless you have internal character data (e.g. created programmatically). Also you should supply a system id, as resolving external entities does not work. The loader from files is much more broken, it always open the file as a Reader and then passes it to InputSource. Instead it should point filename directly to InputSource.

This issue will fix it in trunk and use InputSource in Solr, but will still supply the Reader possibility in previous versions (deprecated).

Issue Links

• incorporates: LUCENE-2731 HyphenationCompoundWordTokenFilter fails to load DTD in Crimson parser (JDK 1.4)