Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-2038

A more accurate facility for detecting Charset Encoding of HTML documents

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • core, detector
    • None

    Description

      Currently, Tika uses icu4j for detecting charset encoding of HTML documents as well as the other naturally text documents. But the accuracy of encoding detector tools, including icu4j, in dealing with the HTML documents is meaningfully less than from which the other text documents. Hence, in our project I developed a library that works pretty well for HTML documents, which is available here: https://github.com/shabanali-faghani/IUST-HTMLCharDet

      Since Tika is widely used with and within some of other Apache stuffs such as Nutch, Lucene, Solr, etc. and these projects are strongly in connection with the HTML documents, it seems that having such an facility in Tika also will help them to become more accurate.

      Attachments

        1. comparisons_20160803b.xlsx
          104 kB
          Tim Allison
        2. comparisons_20160804.xlsx
          115 kB
          Tim Allison
        3. iust_encodings.zip
          7 kB
          Tim Allison
        4. lang-wise-eval_results.zip
          2.51 MB
          Shabanali Faghani
        5. lang-wise-eval_runnable.zip
          19.23 MB
          Shabanali Faghani
        6. lang-wise-eval_source_code.zip
          9.53 MB
          Shabanali Faghani
        7. proposedTLDSampling.csv
          1 kB
          Tim Allison
        8. tika_1_14-SNAPSHOT_encoding_detector.zip
          5 kB
          Tim Allison
        9. tld_text_html_plus_H_column.xlsx
          14 kB
          Shabanali Faghani
        10. tld_text_html.xlsx
          13 kB
          Tim Allison

        Issue Links

          contains

          Bug - A problem which impairs or prevents the functions of the product. TIKA-2050 HTMLEncodingDetector class fails on some HTML documents

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-2267 Add common tokens files for tika-eval

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. TIKA-2771 enableInputFilter() wrecks charset detection for some short html documents

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. TIKA-721 UTF16-LE not detected

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-2484 Improve CharsetDetector to recognize UTF-16LE/BE,UTF-32LE/BE and UTF-7 with/without BOMs correctly

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-2940 Consider an ensemble charset detection method

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-539 Encoding detection is too biased by encoding in meta tag

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Reopened

          Task - A task that needs to be done. TIKA-2750 Update regression corpus

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              faghani Shabanali Faghani
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: