Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-3147

Strip punctuation in lang id component within tika-eval

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.25
    • None
    • None

    Description

      I noticed that "the quick brown fox jumped over the lazy dog" was identified as English in tika-eval. However, if I added semi-colons, it was identified as Chinese.

      This is in alignment with what I've recently seen on a new batch of technical documents that are mostly numbers and abbreviations...these are being identified as Chinese.

      Ideally, we'd strip out punctuation while building the models, and while running language id.

      As a short term fix, we should at least strip out punctuation during detection.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tallison Tim Allison
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment