Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5202

LookaheadTokenFilter consumes an extra token in nextToken

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.3.1
    • Fix Version/s: 4.5, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      This is a bit hard to explain except by looking at the test case. I've coded a filter that uses LookaheadTokenFilter. The incrementToken method peeks some tokens. Then, it seems, nextToken in the Lookahead class calls peekToken itself, which seems to me to consume a token so that it's not seen when the derived class sets out to process the next set of tokens.

      In passing, this test case can be used to demonstrate that it does not work to try to use the afterPosition method to set up attributes of the token that we're 'after'. Probably that was never intended. However, I'm hoping for some feedback as to whether the rest of the structure here is as intended for subclasses of LookaheadTokenFilter.

        Attachments

        1. LUCENE-5202.patch
          9 kB
          Benson Margulies
        2. LUCENE-5202.patch
          9 kB
          Michael McCandless

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bmargulies Benson Margulies
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: