Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-2937

Improve legacy HTML charset detector by replicating Standard's behavior for UTF-16

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Our newer standards based detector follows the standard and automatically modifies a charset header of "UTF-16" (and some variants) to "UTF-8". This appears to help quite a bit.

      LegacyHTMLDetector StandardHTMLDetector NumCommonTokensLegacy NumCommonTokensStandard Delta
      UTF-16LE UTF-8 10 27351 27341
      UTF-16BE UTF-8 14 43565 43551
      UTF-16 UTF-8 6099 748024 741925

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-2933 Revisit "replacement" encoding mappings in StandardHtmlEncodingDetector.

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-2936 The stricter StandardHtmlDetector extracts some header charsets where our legacy detector doesn't

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-2940 Consider an ensemble charset detection method

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              Unassigned Unassigned
              tallison Tim Allison
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: