Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-3766

Remove/deprecate Tokenizer's default ctor

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I was working on a new Tokenizer... and I accidentally forgot to call super(input) (and super.reset(input) from my reset method)... which then meant my correctOffset() calls were silently a no-op; this is very trappy.

      Fortunately the awesome BaseTokenStreamTestCase caught this (I hit failures because the offsets were not in fact being corrected).

      One minimal thing we can do (but it sounds like from Robert there may be reasons why we can't) is add assert input != null in Tokenizer.correctOffset:

      Index: lucene/core/src/java/org/apache/lucene/analysis/Tokenizer.java
      ===================================================================
      --- lucene/core/src/java/org/apache/lucene/analysis/Tokenizer.java	(revision 1242316)
      +++ lucene/core/src/java/org/apache/lucene/analysis/Tokenizer.java	(working copy)
      @@ -82,6 +82,7 @@
          * @see CharStream#correctOffset
          */
         protected final int correctOffset(int currentOff) {
      +    assert input != null: "subclass failed to call super(Reader) or super.reset(Reader)";
           return (input instanceof CharStream) ? ((CharStream) input).correctOffset(currentOff) : currentOff;
         }
      

      But best would be to remove the default ctor that leaves input null...

        Attachments

        1. LUCENE-3766.patch
          38 kB
          Robert Muir
        2. LUCENE-3766.patch
          19 kB
          Michael McCandless

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mikemccand Michael McCandless
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: