Solr
  1. Solr
  2. SOLR-2909

Add support for ResourceLoaderAware tokenizerFactories in synonym filter factories

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.4, 4.0-ALPHA
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: Schema and Analysis
    • Labels:
      None

      Description

      The optional custom tokenizerFactory in SlowSynonymFilterFactory and FSTSynonymFilterFactory might require the ResourceLoader information. Thus inform(ResourceLoader) should be called if the specified tokenizerFactory is an instance of ResourceLoaderAware.

      private static TokenizerFactory loadTokenizerFactory(ResourceLoader loader, String cname, Map<String, String> args) {
        TokenizerFactory tokFactory = (TokenizerFactory) loader.newInstance(cname);
        tokFactory.init(args);
        if (tokFactory instanceof ResourceLoaderAware) {
          ((ResourceLoaderAware) tokFactory).inform(loader);
        }
        return tokFactory;
      }
      
      1. SOLR-2909.patch
        2 kB
        Koji Sekiguchi

        Activity

        Hide
        Jun Ohtani added a comment -

        A problem seems to happen when using TokenizerFactory of Kuromoji.
        KuromojiTokenizerFactory load userDictionaryPath and mode in inform().

        Show
        Jun Ohtani added a comment - A problem seems to happen when using TokenizerFactory of Kuromoji. KuromojiTokenizerFactory load userDictionaryPath and mode in inform().
        Hide
        Koji Sekiguchi added a comment -

        I've talked with Ohtani-san for this issue. He hit this problem and got NPE when using another JapaneseTokenizer (lucene-gosen) with SynonymFilterFactory.

        As I agree with him about that the patch looks nice provided in the Description section above, I'd like to commit if nobody objects.

        Show
        Koji Sekiguchi added a comment - I've talked with Ohtani-san for this issue. He hit this problem and got NPE when using another JapaneseTokenizer (lucene-gosen) with SynonymFilterFactory. As I agree with him about that the patch looks nice provided in the Description section above, I'd like to commit if nobody objects.
        Hide
        Christian Moen added a comment -

        Ohtani-san, good catch. Sekiguchi-san, this sounds like a good idea to me.

        My understanding is that the problem also occurs when using SynonymFilterFactory and that the above will address this case as well since the actual synonym handling is being done by SlowSynonymFilterFactory or FSTSynonymFilterFactory through delegation (based on version). Is this correct? Thanks.

        Show
        Christian Moen added a comment - Ohtani-san, good catch. Sekiguchi-san, this sounds like a good idea to me. My understanding is that the problem also occurs when using SynonymFilterFactory and that the above will address this case as well since the actual synonym handling is being done by SlowSynonymFilterFactory or FSTSynonymFilterFactory through delegation (based on version). Is this correct? Thanks.
        Hide
        Jun Ohtani added a comment -

        Yes.
        Probably the word of the user's dictionary is not used in SynonymFilter.

        Show
        Jun Ohtani added a comment - Yes. Probably the word of the user's dictionary is not used in SynonymFilter.
        Hide
        Christian Moen added a comment -

        Thanks.

        Sekiguchi-san, I'm happy to test and review a patch if you'd like that.

        Show
        Christian Moen added a comment - Thanks. Sekiguchi-san, I'm happy to test and review a patch if you'd like that.
        Hide
        Robert Muir added a comment -

        +1, good catch.

        Show
        Robert Muir added a comment - +1, good catch.
        Hide
        Koji Sekiguchi added a comment -

        trunk: Committed revision 1291541.
        branch_3x: Committed revision 1291543.

        Thanks, everyone!

        Show
        Koji Sekiguchi added a comment - trunk: Committed revision 1291541. branch_3x: Committed revision 1291543. Thanks, everyone!

          People

          • Assignee:
            Koji Sekiguchi
            Reporter:
            Tom Klonikowski
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development