Lucene - Core
  1. Lucene - Core
  2. LUCENE-3064

add checks to MockTokenizer to enforce proper consumption

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      we can enforce things like consumer properly iterates through tokenstream lifeycle
      via MockTokenizer. this could catch bugs in consumers that don't call reset(), etc.

      1. LUCENE-3064.patch
        30 kB
        Robert Muir
      2. LUCENE-3064.patch
        27 kB
        Robert Muir
      3. LUCENE-3064.patch
        18 kB
        Robert Muir
      4. LUCENE-3064.patch
        9 kB
        Robert Muir

        Activity

        Hide
        Robert Muir added a comment -

        a first stab at a patch, i got the core tests passing but it still be better, and there are problems in contrib.

        Show
        Robert Muir added a comment - a first stab at a patch, i got the core tests passing but it still be better, and there are problems in contrib.
        Hide
        Robert Muir added a comment -

        updated patch with fixes for contrib, though highlighter still remains, and some TODOs are not resolved.

        Show
        Robert Muir added a comment - updated patch with fixes for contrib, though highlighter still remains, and some TODOs are not resolved.
        Hide
        Robert Muir added a comment -

        Updated patch: I think this is ready to commit.

        I added a boolean to allow the workflow checks to be disabled in very exceptional cases (e.g. TestIndexWriterExceptions's CrashingTokenFilter), so in general we can do pretty good checking.

        Show
        Robert Muir added a comment - Updated patch: I think this is ready to commit. I added a boolean to allow the workflow checks to be disabled in very exceptional cases (e.g. TestIndexWriterExceptions's CrashingTokenFilter), so in general we can do pretty good checking.
        Hide
        Robert Muir added a comment -

        Sorry, in the previous patch I had disabled the checks accidentally by default in MockAnalyzer!

        Show
        Robert Muir added a comment - Sorry, in the previous patch I had disabled the checks accidentally by default in MockAnalyzer!
        Hide
        Robert Muir added a comment -

        Committed revision 1102290.

        I know the checks aren't totally perfect, but I think this is an important step to switching all of our APIs over to reusableTokenStream (LUCENE-3040), and ultimately looking at a way to do something like deprecate tokenStream.

        For this reason I will now look at backporting to 3.x

        Show
        Robert Muir added a comment - Committed revision 1102290. I know the checks aren't totally perfect, but I think this is an important step to switching all of our APIs over to reusableTokenStream ( LUCENE-3040 ), and ultimately looking at a way to do something like deprecate tokenStream. For this reason I will now look at backporting to 3.x
        Hide
        Robert Muir added a comment -

        backported to 3.x in revision 1102812

        Show
        Robert Muir added a comment - backported to 3.x in revision 1102812
        Hide
        Robert Muir added a comment -

        Bulk closing for 3.2

        Show
        Robert Muir added a comment - Bulk closing for 3.2

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development