Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-3147

Strip punctuation in lang id component within tika-eval

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.25
    • Component/s: None
    • Labels:
      None

      Description

      I noticed that "the quick brown fox jumped over the lazy dog" was identified as English in tika-eval. However, if I added semi-colons, it was identified as Chinese.

      This is in alignment with what I've recently seen on a new batch of technical documents that are mostly numbers and abbreviations...these are being identified as Chinese.

      Ideally, we'd strip out punctuation while building the models, and while running language id.

      As a short term fix, we should at least strip out punctuation during detection.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tallison Tim Allison
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: