[OPENNLP-862] BRAT format packages do not handle punctuation correctly when training NER model

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.6.0
• Component/s: Formats
• Assignee: Jörn Kottmann
• Reporter: Gregory Werner
• Created: 28/Sep/16 13:32
• Updated: 25/Jan/17 14:47

Description

BRAT does not require preprocessing of text files in order to add annotations to text documents. And this is great because I can feed documents from corpora I am given directly into BRAT. If I have a line such as:

Residence: Athens, Georgia

I would provide 2 annotations in BRAT, Athens and Georgia, and BRAT would generate the offset and everything would be fine.

It appears though that I only get 1 entity correctly processed (and the other dropped) in OpenNLP with TokenNameFinderTrainer.brat, Georgia, because the comma is not separated from Athens. I have 789 annotated raw, non pre-processed text documents from past efforts. I believe that OpenNLP should be able to handle lines like the above in the case of the BRAT format code.

It appears that BratNameSampleStream uses the WhitespaceTokenizer and that is what creates Athens, as a token. I find that the SimpleTokenizer might perform better with BRAT through my limited testing of raw documents if the current general approach is held.