Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-1696

Language Identification with Text Processing Toolkit from MITLL

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.13
    • languageidentifier
    • None

    Description

      The aim here is to extend the methods for language identification within text. MIT Lincoln Labs has an open source library [1] written in Julia. Having spoken with the MITLL guys there is a possibility that there is a scala version of this library which would make it easier to package in with Tika.

      At this point I'm not quite sure how many languages this library supports by default but it can be extended when provided some training data.

      [1] https://github.com/mit-nlp/Text.jl

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-1723 Integrate language-detector into Tika

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is required by

          Improvement - An improvement or enhancement to an existing feature or task. TIKA-1872 Backport tika-langdetect from 2.x branch to 1.13 branch

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              chrismattmann Chris A. Mattmann
              pramirez Paul Ramirez
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: