[OPENNLP-471] DictionaryNameFinder has HASHing issues

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: tools-1.5.3
• Component/s: Name Finder
• Labels: dictionary namefinder
• Assignee: James Kosin
• Reporter: James Kosin
• Created: 15/Mar/12 03:47
• Updated: 25/Apr/12 01:54
• Resolved: 25/Apr/12 01:54

Description

The DictionaryNameFinder has issues finding multi-token names when the dictionary is searched a token at a time by the find() method. If, the dictionary doesn't have a single (or shorter) token match available in the dictionary.

Having a dictionary with

{"folic", "acid"}

without an entry for

{"folic"}

will cause the find() method to totally skip the fact there is a longer match possible.

Thanks to Jim for pushing this and to my debugging skills to find.

Two possiblilites come to mind:
1) I don't really like, is we turn it into a larger problem by trying longer matches when shorter ones don't match. Unfortunately, this turns quickly into a race to see who can wait longer.

2) A way of returning a possible match that may need exploring, or a look-ahead type system to say we don't match "folic" but if you have "acid" after "folic" we have a match for that in the dictionary.

3) Leave it as is and modify the dictionary to add shorter terms to the dictionary... maybe marking as not-a-valid entry so we can know we need a longer match.