Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-3113

fix analyzer bugs found by MockTokenizer

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2, 4.0-ALPHA
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      In LUCENE-3064, we beefed up MockTokenizer with assertions, and I've switched over the analysis tests to use MockTokenizer for better coverage.

      However, this found a few bugs (one of which is LUCENE-3106):

      • incrementToken() after it returns false in CommonGramsQueryFilter, HyphenatedWordsFilter, ShingleFilter, SynonymFilter
      • missing end() implementation for PrefixAwareTokenFilter
      • double reset() in QueryAutoStopWordAnalyzer and ReusableAnalyzerBase
      • missing correctOffset()s in MockTokenizer itself.

      I think it would be nice to just fix all the bugs on one issue... I've fixed everything except Shingle and Synonym

        Attachments

        1. LUCENE-3113.patch
          113 kB
          Robert Muir
        2. LUCENE-3113.patch
          109 kB
          Robert Muir

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rcmuir Robert Muir
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: