[CTAKES-372] Penn TreeBank Tokenizer could use some attention

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: future enhancement
• Component/s: ctakes-core
• Labels: features performance
• Assignee: James Joseph Masanz
• Reporter: Sean Finan
• Created: 23/Jul/15 14:58
• Updated: 06/Mar/17 16:19

Description

The ptb tokenizer currently in use by ctakes has some inconsistencies. See https://issues.apache.org/jira/browse/CTAKES-371 It also does not seem to incorporate some of the clinical rules set out in http://clear.colorado.edu/compsem/documents/treebank_guidelines.pdf

Some major refactoring is also in order ... as are numerous test cases.