Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8250

Should FilteringTokenFilter handle positionLength

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      FilteringTokenFilter does not handle the position length graph attribute when removing a token from the stream. This doesn't work well with graph token stream that sets position length since removing a token from the stream can invalidate the position length set on the previous tokens.
      This issue was first discussed in https://issues.apache.org/jira/browse/LUCENE-4065 but it has a different purpose which is why I am opening a new issue here.

        Attachments

        1. LUCENE-8250.patch
          3 kB
          Jim Ferenczi

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jim.ferenczi Jim Ferenczi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: