Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-3612

Update StandardHtmlEncodingDetector to follow the living standard

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.1.0
    • None
    • detector
    • None

    Description

      StandardHtmlEncodingDetector uses 3 heuristics to detect the encoding of a HTML document:

      1. BOM
      2. Content-Type HTTP header
      3. HTML <meta> tag

      The "living standard", 13.2.3.2 Determining the character encoding has evolved since then and is based one a longer chain of steps/approaches, including char/byte statistics ("The user agent may attempt to autodetect the character encoding from applying frequency analysis or other algorithms to the data stream.") and (definitely useful) a list of fall-back encodings based on the content language and if the document is not encoded using one of the UTF encodings.

      Attachments

        Activity

          People

            Unassigned Unassigned
            snagel Sebastian Nagel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: