[LUCENE-589] Demo HTML parser doesn't work for international documents

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/examples
• Assignee: Robert Muir
• Reporter: Curtis d'Entremont
• Created: 07/Jun/06 22:25
• Updated: 28/Aug/22 11:28
• Resolved: 05/Nov/10 07:45

Description

Javacc assumes ASCII so it won't work with, say, japanese documents. Ideally it would read the charset from the HTML markup, but that can by tricky. For now assuming unicode would do the trick:

Add the following line marked with a + to HTMLParser.jj:

options {
STATIC = false;
OPTIMIZE_TOKEN_MANAGER = true;
//DEBUG_LOOKAHEAD = true;
//DEBUG_TOKEN_MANAGER = true;
+ UNICODE_INPUT = true;
}