Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
4.0.0
-
None
-
None
Description
SentenceDetectorAnnotatorBIO iterates over every character in the Segment and classifies it as Begin, Inside, or Outside a Sentence. When doing this, it needs to know the next and previous token from the current character.
It currently finds these tokens afresh for each character. That means that it starts from the current character, and scans forward and backwards looking for whitespace until it finds the boundaries of the tokens either side of the current position. This is very wasteful; when the current index steps within a word, the tokens do not change since we're still within the same word. Also, since we're scanning in one direction, we never need to scan for the previous token, because we already know it.
(I found this bug with a pathological case where I had a "document" with a single word that was a megabyte long. In a case where the word length is not bounded, the current algorithm is quadratic instead of linear, because it scans the length of the word for each character.)
Patch attached. This fixes the problem by keeping track of the word boundary, and only scanning for the next token when we have reached the boundary of the current one. Also, the previous token is simply taken as the token from the previous iteration, and the token features are only recomputed when the token changes.