Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-2592

HTML with charset unicode handled as utf-16 instead utf-8

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0, 1.16, 1.17
    • Fix Version/s: 1.18, 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      HTML files are detected as utf-16 when meta content is set to "unicode".

      <meta http-equiv="Content-Type" content="text/html; charset="unicode">
       

       
      Shouldn't the default be utf-8?

      The attached sample file is shown correctly in:
      Chromium Version 55.0.2883.75
      Firefox 50.1.0
      IE 11

      I am aware that there is no charset "unicode" (available character encodings: http://www.iana.org/assignments/character-sets/character-sets.xhtml)

      Unfortunately there are many wrong encodings used out there.

      All unknown encodings should be validated or at least be set to default utf-8.

      Regards

      Andreas

        Attachments

        1. TestHTMLCharsetCP1256.html
          0.7 kB
          Andreas Meier
        2. TestHTMLCharsetArabicCP1256.html
          0.7 kB
          Andreas Meier
        3. TestCharsetUnicodeHTML.html
          0.2 kB
          Andreas Meier
        4. StandardCharsets_unsupported_by_IANA.txt
          1 kB
          Andreas Meier
        5. IANA Charset names.txt
          10 kB
          Ken Krugler
        6. fix-for-TIKA2592-contributed-by-Andreas-Meier.patch
          1 kB
          Andreas Meier

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                AndreasMeier Andreas Meier
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: