Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-2671

HtmlEncodingDetector doesnt take provided metadata into account

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: detector
    • Labels:
      None

      Description

      org.apache.tika.parser.html.HtmlEncodingDetector ignores the document's metadata. So when using it to detect the charset of an HTML document that came with a conflicting charset specified at the transport layer level, the encoding specified inside the file is used instead.

      This behavior does not conform to what is specified by the W3C for determining the character encoding of HTML pages. This causes bugs similar to NUTCH-2599.

      If HtmlEncodingDetector is not meant to take into account meta-information about the document, then maybe another detector should be provided, that would be a CompositeDetector including, in that order:

      • a new, simple, MetadataEncodingDetector, that would simply return the encoding
      • the existing HtmlEncodingDetector
      • a generic detector, like UniversalEncodingDetector

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                gbouchar Gerard Bouchar
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: