[JOSHUA-326] Make preprocessing phase pluggable

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 6.2
• Component/s: pipeline
• Assignee: Tommaso Teofili
• Reporter: Tommaso Teofili
• Created: 22/Dec/16 14:42
• Updated: 22/Dec/16 15:21

Description

It'd be nice to have the data preprocessing phase pluggable, with a default simple Java implementation and eventually other more advanced ones based on external tools like Apache OpenNLP.
That should replace our scripts based preprocessing:

• tokenization: https://github.com/apache/incubator-joshua/blob/master/scripts/preparation/tokenize.pl