[OPENNLP-830] Huge runtime improvement on training (POS, Chunk, ...)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.0
• Component/s: Machine Learning, POS Tagger
• Labels: performance
• Environment: Any
• Flags: Patch, Important
• Assignee: Jörn Kottmann
• Reporter: Julien Subercaze
• Created: 16/Jan/16 22:55
• Updated: 28/Oct/16 15:55
• Resolved: 27/Oct/16 20:25

Description

opennlp.tools.ml.model.IndexHashTable is custom-made Hashtable that is used to store mapping index. This Hashtable is heavily used in openlp.tools.ml.* (i.e. every model) and leads to disastrous performance.

This hashtable is probably legacy some legacy and is highly inefficient. A simple drop-in replacement by a java.util.HashMap wrapper solves the issue, doesn't break compatibility and does not add any dependency.

Training a pos-tagger on a large dataset with custom tags, I see a factor 5 improvement. It also seems to improve all ML models training pipeline.

See : https://github.com/jsubercaze/opennlp/blob/trunk/opennlp-tools/src/main/java/opennlp/tools/ml/model/IndexHashTable.java

For a quick fix.