[CTAKES-143] dictionary lookup iterates inefficiently

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: ctakes-dictionary-lookup
• Assignee: Unassigned
• Reporter: Tim Miller
• Created: 04/Feb/13 21:33
• Updated: 06/Feb/13 14:27
• Resolved: 06/Feb/13 14:27

Description

I noticed quadratic performance on large notes (in length of file) mainly due to dictionary annotator. Inside DictionaryLookupAnnotator, it iterates over "Lookup windows." Inside this iteration, it calls getLookupTokenIterator() in the LookupInitializer. As implemented in FirstTokenPermLookupInitializerImpl, this method will create a iterator over all BaseTokens, pruning out certain subtypes. This is then passed back to the DictionaryLookupAnnotator, which will only then constrain the list to those tokens which overlap the lookup window. This iterating over every token in the document for each lookup window is extremely inefficient. This could be fixed by, e.g., doing the constraining first and then doing the pruning by sub-types.