Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5191

SimpleHTMLEncoder in Highlighter module breaks Unicode outside BMP

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 4.5, 6.0
    • modules/highlighter
    • None
    • New

    Description

      The highlighter provides a function to escape HTML, which does to much. To create valid HTML only ", <, >, & must be escaped, everything else can kept unescaped. The escaper unfortunately does also additionally escape everything > 127, which is unneeded if your web site has the correct encoding. It also produces huge amounts of HTML entities if used with eastern languages.

      This would not be a bugf if the escaping would be correct, but it isn't, it escapes like that:

      result.append("\&#").append((int)ch).append(";");

      So it escapes not (as HTML needs) the unicode codepoint, instead it escapes the UTF-16 char, which is incorrect, e.g. for our all-time favourite Deseret:

      U+10400 (deseret capital letter long i) would be escaped as &#55297;&#56320; and not as &#66560;.

      So we should remove the stupid encoding of chars > 127 which is simply useless

      See also: https://github.com/elasticsearch/elasticsearch/issues/3587

      Attachments

        1. LUCENE-5191.patch
          2 kB
          Uwe Schindler
        2. LUCENE-5191.patch
          0.7 kB
          Uwe Schindler

        Activity

          People

            uschindler Uwe Schindler
            uschindler Uwe Schindler
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: