Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
New
Description
If we have a set of paths of differing lengths at the end of a TokenStream, FixedShingleFilter will currently stop when the first path hits the end of the stream, but this may miss extra longer paths. For example, the stream "a b:3/c d e f" with a shingle size of 3 should produce paths "a b f", "a c d", "c d e" and "d e f", but at the moment FixedShingleFilter will stop when it finds that the path "b f" is too short to produce a shingle.