Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-3848

basetokenstreamtestcase should fail if tokenstream starts with posinc=0

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.6, 4.0-ALPHA
    • None
    • None
    • New

    Description

      This is meaningless for a tokenstream to start with posinc=0,

      Its also caused problems and hairiness in the indexer (LUCENE-1255, LUCENE-1542),
      and it makes senseless tokenstreams. We should add a check and fix any that do this.

      Furthermore the same bug can exist in removing-filters if they have enablePositionIncrements=false.
      I think this option is useful: but it shouldnt mean 'allow broken tokenstream', it just means we
      don't add gaps.

      If you remove tokens with enablePositionIncrements=false it should not cause the TS to start with
      positionincrement=0, and it shouldnt 'restructure' the tokenstream (e.g. moving synonyms on top of a different word).
      It should just not add any 'holes'.

      Attachments

        1. LUCENE-3848.patch
          2 kB
          Robert Muir
        2. LUCENE-3848-MockGraphTokenFilter.patch
          21 kB
          Michael McCandless
        3. LUCENE-3848.patch
          9 kB
          Robert Muir

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            rcmuir Robert Muir
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment