Details

      Description

      Implement an separate OpenNLP Tokenizer Engine.

      While some Engines like the OpenNLP POS or the CELI Lemmatizer engine do support tokenizing (if tokens do not already exist in the Analyzed Text) it is important to implement an engine explicitly for this task.

      This engine also supports the language configuration (see following example)

      en;model=SIMPLE
      de;model=mySpecificTokenizerModel_de.bin
      !jp
      !zh
      *

      the 'model' parameter can be used to load specific tokenizer models. "SIMPLE" forces the use of the OpenNLP SimpleTokenizer. If no model configuration is present the default tokenizer for the language is loaded ("

      {lang}

      -token.bin" or the simple tokenizer if the language model is not present).

        Attachments

          Activity

            People

            • Assignee:
              rwesten Rupert Westenthaler
              Reporter:
              rwesten Rupert Westenthaler
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: