Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5191

SimpleHTMLEncoder in Highlighter module breaks Unicode outside BMP

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.5, 6.0
    • Component/s: modules/highlighter
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The highlighter provides a function to escape HTML, which does to much. To create valid HTML only ", <, >, & must be escaped, everything else can kept unescaped. The escaper unfortunately does also additionally escape everything > 127, which is unneeded if your web site has the correct encoding. It also produces huge amounts of HTML entities if used with eastern languages.

      This would not be a bugf if the escaping would be correct, but it isn't, it escapes like that:

      result.append("\&#").append((int)ch).append(";");

      So it escapes not (as HTML needs) the unicode codepoint, instead it escapes the UTF-16 char, which is incorrect, e.g. for our all-time favourite Deseret:

      U+10400 (deseret capital letter long i) would be escaped as &#55297;&#56320; and not as &#66560;.

      So we should remove the stupid encoding of chars > 127 which is simply useless

      See also: https://github.com/elasticsearch/elasticsearch/issues/3587

        Attachments

        1. LUCENE-5191.patch
          0.7 kB
          Uwe Schindler
        2. LUCENE-5191.patch
          2 kB
          Uwe Schindler

          Activity

            People

            • Assignee:
              thetaphi Uwe Schindler
              Reporter:
              thetaphi Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: