Lucene - Core
  1. Lucene - Core
  2. LUCENE-3848

basetokenstreamtestcase should fail if tokenstream starts with posinc=0

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      This is meaningless for a tokenstream to start with posinc=0,

      Its also caused problems and hairiness in the indexer (LUCENE-1255, LUCENE-1542),
      and it makes senseless tokenstreams. We should add a check and fix any that do this.

      Furthermore the same bug can exist in removing-filters if they have enablePositionIncrements=false.
      I think this option is useful: but it shouldnt mean 'allow broken tokenstream', it just means we
      don't add gaps.

      If you remove tokens with enablePositionIncrements=false it should not cause the TS to start with
      positionincrement=0, and it shouldnt 'restructure' the tokenstream (e.g. moving synonyms on top of a different word).
      It should just not add any 'holes'.

      1. LUCENE-3848.patch
        9 kB
        Robert Muir
      2. LUCENE-3848-MockGraphTokenFilter.patch
        21 kB
        Michael McCandless
      3. LUCENE-3848.patch
        2 kB
        Robert Muir

        Activity

        Robert Muir created issue -
        Robert Muir made changes -
        Field Original Value New Value
        Attachment LUCENE-3848.patch [ 12517016 ]
        Michael McCandless made changes -
        Attachment LUCENE-3848-MockGraphTokenFilter.patch [ 12517022 ]
        Robert Muir made changes -
        Fix Version/s 3.6 [ 12319070 ]
        Robert Muir made changes -
        Attachment LUCENE-3848.patch [ 12518482 ]
        Robert Muir made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 3.6 [ 12319070 ]
        Resolution Fixed [ 1 ]
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development