Lucene - Core
  1. Lucene - Core
  2. LUCENE-5908

Assertion fails for Lucene43NGramTokenizer

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      As a side effect from LUCENE-5859, Lucene43NGramTokenizer was made not final. This can trip an assert we have that Tokenizer need to either be final, or have their incrementToken() function be final.

      1. LUCENE-5908.patch
        11 kB
        Ryan Ernst
      2. LUCENE-5908.patch
        2 kB
        Ryan Ernst

        Activity

        Hide
        Ryan Ernst added a comment -

        Simple patch.

        Show
        Ryan Ernst added a comment - Simple patch.
        Hide
        Uwe Schindler added a comment -

        Why is the Tokenizer itsself not final. I looked into the history, you removed the final keyword, so it was final before.

        Was there a reason to remove it?

        Show
        Uwe Schindler added a comment - Why is the Tokenizer itsself not final. I looked into the history, you removed the final keyword, so it was final before. Was there a reason to remove it?
        Hide
        Ryan Ernst added a comment -

        IIRC, I was adding Lucene43EdgeNGramTokenizer, which was almost exactly the same, and didn't want to duplicate it. Here I just went the route that satisfies the assert. If you feel strongly, I can copy the impl and make it final.

        Show
        Ryan Ernst added a comment - IIRC, I was adding Lucene43EdgeNGramTokenizer , which was almost exactly the same, and didn't want to duplicate it. Here I just went the route that satisfies the assert. If you feel strongly, I can copy the impl and make it final.
        Hide
        Uwe Schindler added a comment -

        In my 4.10 checkout it looks like that:

        public final class Lucene43EdgeNGramTokenizer extends Tokenizer {
        public class Lucene43NGramTokenizer extends Tokenizer {
        

        So it does not relate to each other.

        I know that EdgeNGramTokenizer extends NGramTokenizer, which explains the non-finalness.

        Show
        Uwe Schindler added a comment - In my 4.10 checkout it looks like that: public final class Lucene43EdgeNGramTokenizer extends Tokenizer { public class Lucene43NGramTokenizer extends Tokenizer { So it does not relate to each other. I know that EdgeNGramTokenizer extends NGramTokenizer, which explains the non-finalness.
        Hide
        Uwe Schindler added a comment -

        Ah I see: In 4.x its code duplication, in trunk it extends each other. We should fix that, too. But I am nervous about trunk, because it has no tests.

        Show
        Uwe Schindler added a comment - Ah I see: In 4.x its code duplication, in trunk it extends each other. We should fix that, too. But I am nervous about trunk, because it has no tests.
        Hide
        Ryan Ernst added a comment -

        Ok, this new patch replicates branch_4x's Lucene43EdgeNGramTokenizer to trunk (minus the diffs like Reader and Version that are no longer there in trunk). I think this should make the backport easy (so it will just be making Lucene43NGramTokenizer final).

        Show
        Ryan Ernst added a comment - Ok, this new patch replicates branch_4x's Lucene43EdgeNGramTokenizer to trunk (minus the diffs like Reader and Version that are no longer there in trunk). I think this should make the backport easy (so it will just be making Lucene43NGramTokenizer final).
        Hide
        ASF subversion and git services added a comment -

        Commit 1620759 from Ryan Ernst in branch 'dev/trunk'
        [ https://svn.apache.org/r1620759 ]

        LUCENE-5908: Fix Lucene43NGramTokenizer to be final

        Show
        ASF subversion and git services added a comment - Commit 1620759 from Ryan Ernst in branch 'dev/trunk' [ https://svn.apache.org/r1620759 ] LUCENE-5908 : Fix Lucene43NGramTokenizer to be final
        Hide
        ASF subversion and git services added a comment -

        Commit 1620760 from Ryan Ernst in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1620760 ]

        LUCENE-5908: Fix Lucene43NGramTokenizer to be final (merge 1620759)

        Show
        ASF subversion and git services added a comment - Commit 1620760 from Ryan Ernst in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1620760 ] LUCENE-5908 : Fix Lucene43NGramTokenizer to be final (merge 1620759)
        Hide
        ASF subversion and git services added a comment -

        Commit 1620762 from Ryan Ernst in branch 'dev/branches/lucene_solr_4_10'
        [ https://svn.apache.org/r1620762 ]

        LUCENE-5908: Backport to 4.10 (merged 1620760)

        Show
        ASF subversion and git services added a comment - Commit 1620762 from Ryan Ernst in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1620762 ] LUCENE-5908 : Backport to 4.10 (merged 1620760)

          People

          • Assignee:
            Ryan Ernst
            Reporter:
            Ryan Ernst
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development