Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-2909

Add support for ResourceLoaderAware tokenizerFactories in synonym filter factories

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.4, 4.0-ALPHA
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: Schema and Analysis
    • Labels:
      None

      Description

      The optional custom tokenizerFactory in SlowSynonymFilterFactory and FSTSynonymFilterFactory might require the ResourceLoader information. Thus inform(ResourceLoader) should be called if the specified tokenizerFactory is an instance of ResourceLoaderAware.

      private static TokenizerFactory loadTokenizerFactory(ResourceLoader loader, String cname, Map<String, String> args) {
        TokenizerFactory tokFactory = (TokenizerFactory) loader.newInstance(cname);
        tokFactory.init(args);
        if (tokFactory instanceof ResourceLoaderAware) {
          ((ResourceLoaderAware) tokFactory).inform(loader);
        }
        return tokFactory;
      }
      
      1. SOLR-2909.patch
        2 kB
        Koji Sekiguchi

        Activity

        Hide
        koji Koji Sekiguchi added a comment -

        trunk: Committed revision 1291541.
        branch_3x: Committed revision 1291543.

        Thanks, everyone!

        Show
        koji Koji Sekiguchi added a comment - trunk: Committed revision 1291541. branch_3x: Committed revision 1291543. Thanks, everyone!
        Hide
        rcmuir Robert Muir added a comment -

        +1, good catch.

        Show
        rcmuir Robert Muir added a comment - +1, good catch.
        Hide
        cm Christian Moen added a comment -

        Thanks.

        Sekiguchi-san, I'm happy to test and review a patch if you'd like that.

        Show
        cm Christian Moen added a comment - Thanks. Sekiguchi-san, I'm happy to test and review a patch if you'd like that.
        Hide
        jun_o Jun Ohtani added a comment -

        Yes.
        Probably the word of the user's dictionary is not used in SynonymFilter.

        Show
        jun_o Jun Ohtani added a comment - Yes. Probably the word of the user's dictionary is not used in SynonymFilter.
        Hide
        cm Christian Moen added a comment -

        Ohtani-san, good catch. Sekiguchi-san, this sounds like a good idea to me.

        My understanding is that the problem also occurs when using SynonymFilterFactory and that the above will address this case as well since the actual synonym handling is being done by SlowSynonymFilterFactory or FSTSynonymFilterFactory through delegation (based on version). Is this correct? Thanks.

        Show
        cm Christian Moen added a comment - Ohtani-san, good catch. Sekiguchi-san, this sounds like a good idea to me. My understanding is that the problem also occurs when using SynonymFilterFactory and that the above will address this case as well since the actual synonym handling is being done by SlowSynonymFilterFactory or FSTSynonymFilterFactory through delegation (based on version). Is this correct? Thanks.
        Hide
        koji Koji Sekiguchi added a comment -

        I've talked with Ohtani-san for this issue. He hit this problem and got NPE when using another JapaneseTokenizer (lucene-gosen) with SynonymFilterFactory.

        As I agree with him about that the patch looks nice provided in the Description section above, I'd like to commit if nobody objects.

        Show
        koji Koji Sekiguchi added a comment - I've talked with Ohtani-san for this issue. He hit this problem and got NPE when using another JapaneseTokenizer (lucene-gosen) with SynonymFilterFactory. As I agree with him about that the patch looks nice provided in the Description section above, I'd like to commit if nobody objects.
        Hide
        jun_o Jun Ohtani added a comment -

        A problem seems to happen when using TokenizerFactory of Kuromoji.
        KuromojiTokenizerFactory load userDictionaryPath and mode in inform().

        Show
        jun_o Jun Ohtani added a comment - A problem seems to happen when using TokenizerFactory of Kuromoji. KuromojiTokenizerFactory load userDictionaryPath and mode in inform().

          People

          • Assignee:
            koji Koji Sekiguchi
            Reporter:
            klonik_t Tom Klonikowski
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development