Solr
  1. Solr
  2. SOLR-1307

Provide a standard way to reload plugins

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 4.9, 5.0
    • Component/s: search, update
    • Labels:
      None

      Description

      Currently, Solr plugins have no standard way to reload themselves. Each plugin invents its own mechanism e.g. SpellCheckComponent. For others, even small changes to configuration files are visible only after a core reload. Examples include changing elevate.xml, stopwords.txt etc.

      We should provide a standard way for plugins to reload themselves on events relevant to them.

        Issue Links

          Activity

          Hide
          Hoss Man added a comment -

          this could be very hairy depending on what you mean by "reload"

          when we talk about reloading cores, what happens is we instantiate a new core, then replace refrences to the old core with refrences to the new core.

          attempting to do something similar with each and every type of plugin seems like it could get incredibly tedious to deal with the synchronization issues.

          what seems like it might make the most sense for the common case is to have helper code available to make it easier for plugins to do their own internal reloading (ie: instead of constructing a new instance of SynonymFIlterFactory, we make it easy for an instance of SynomymFilterFactory to reload it's own synonym text file). going this route means that only the classes that want to be reloadable need to worry about the synchronization costs associated with doing so (ie: StopWorkFilterFactory can put synchronization on some getStopWordSet() method it uses when constructing each StopFilter instance)

          This is actually something that's already possible now: any plugin can spin up a TimerTask to reload things in a background thread, SolrCoreAware plugins can programaticly register newSearcher listeners to make callbacks, etc... We probably just want to provide helper code to make this easier for people, and add the functionality to some of the obvious choices in the plugins that ship with Solr along with options to enable it (you might want a query analyzer instance of SynonymFilterFactory to md5/reload the synonyms file on every commit, but an index analyzer instance of SYnonymFilterFactory probably shouldn't)

          Show
          Hoss Man added a comment - this could be very hairy depending on what you mean by "reload" when we talk about reloading cores, what happens is we instantiate a new core, then replace refrences to the old core with refrences to the new core. attempting to do something similar with each and every type of plugin seems like it could get incredibly tedious to deal with the synchronization issues. what seems like it might make the most sense for the common case is to have helper code available to make it easier for plugins to do their own internal reloading (ie: instead of constructing a new instance of SynonymFIlterFactory, we make it easy for an instance of SynomymFilterFactory to reload it's own synonym text file). going this route means that only the classes that want to be reloadable need to worry about the synchronization costs associated with doing so (ie: StopWorkFilterFactory can put synchronization on some getStopWordSet() method it uses when constructing each StopFilter instance) This is actually something that's already possible now: any plugin can spin up a TimerTask to reload things in a background thread, SolrCoreAware plugins can programaticly register newSearcher listeners to make callbacks, etc... We probably just want to provide helper code to make this easier for people, and add the functionality to some of the obvious choices in the plugins that ship with Solr along with options to enable it (you might want a query analyzer instance of SynonymFilterFactory to md5/reload the synonyms file on every commit, but an index analyzer instance of SYnonymFilterFactory probably shouldn't)
          Hide
          Shalin Shekhar Mangar added a comment -

          this could be very hairy depending on what you mean by "reload"

          Yeah, I kept it vague on purpose to have this discussion first.

          when we talk about reloading cores, what happens is we instantiate a new core, then replace refrences to the old core with refrences to the new core.

          attempting to do something similar with each and every type of plugin seems like it could get incredibly tedious to deal with the synchronization issues.

          I had thought about a new instance swapping out the old instance. I actually thought that it may be the easiest approach. But then I realized that it may not be a good approach for resource intensive plugins e.g. a SpellCheckComponent managing several dictionaries. One wouldn't want to create two instances each with their own dictionaries. An easier option would be to just have an interface with a reload method which plugins can implement and let them do whatever they need to do. The onus of thread-safety will be on the plugins themselves.

          This is actually something that's already possible now: any plugin can spin up a TimerTask to reload things in a background thread, SolrCoreAware plugins can programaticly register newSearcher listeners to make callbacks, etc... We probably just want to provide helper code to make this easier for people, and add the functionality to some of the obvious choices in the plugins that ship with Solr along with options to enable it (you might want a query analyzer instance of SynonymFilterFactory to md5/reload the synonyms file on every commit, but an index analyzer instance of SYnonymFilterFactory probably shouldn't)

          Yes, it is possible now. Just that everyone does it their own way and some don't at all. DIH has an HTTP API for reload. SpellCheckComponent has reload/build as HTTP API and buildOnCommit/buildOnOptimize configuration. QueryElevationComponent, SynonymFilter have nothing.

          All I want is to have some consistency in implementing and invoking such operations. Look at SpellCheckComponent's SpellCheckListener for the kind of hacks we've had to do to implement these features.

          Perhaps all we need is a better event listener API? I wouldn't want to have complex configuration to register components with certain kinds of events. As we have limited number of events and limited number of components, perhaps we can pass all events into a plugin and let it decide what it wants to do on that event?

          Show
          Shalin Shekhar Mangar added a comment - this could be very hairy depending on what you mean by "reload" Yeah, I kept it vague on purpose to have this discussion first. when we talk about reloading cores, what happens is we instantiate a new core, then replace refrences to the old core with refrences to the new core. attempting to do something similar with each and every type of plugin seems like it could get incredibly tedious to deal with the synchronization issues. I had thought about a new instance swapping out the old instance. I actually thought that it may be the easiest approach. But then I realized that it may not be a good approach for resource intensive plugins e.g. a SpellCheckComponent managing several dictionaries. One wouldn't want to create two instances each with their own dictionaries. An easier option would be to just have an interface with a reload method which plugins can implement and let them do whatever they need to do. The onus of thread-safety will be on the plugins themselves. This is actually something that's already possible now: any plugin can spin up a TimerTask to reload things in a background thread, SolrCoreAware plugins can programaticly register newSearcher listeners to make callbacks, etc... We probably just want to provide helper code to make this easier for people, and add the functionality to some of the obvious choices in the plugins that ship with Solr along with options to enable it (you might want a query analyzer instance of SynonymFilterFactory to md5/reload the synonyms file on every commit, but an index analyzer instance of SYnonymFilterFactory probably shouldn't) Yes, it is possible now. Just that everyone does it their own way and some don't at all. DIH has an HTTP API for reload. SpellCheckComponent has reload/build as HTTP API and buildOnCommit/buildOnOptimize configuration. QueryElevationComponent, SynonymFilter have nothing. All I want is to have some consistency in implementing and invoking such operations. Look at SpellCheckComponent's SpellCheckListener for the kind of hacks we've had to do to implement these features. Perhaps all we need is a better event listener API? I wouldn't want to have complex configuration to register components with certain kinds of events. As we have limited number of events and limited number of components, perhaps we can pass all events into a plugin and let it decide what it wants to do on that event?
          Hide
          Noble Paul added a comment - - edited

          The possible events as I see are config file changes , commit /optimize. commit/optimize are already handled . so the main requirement is to make it possible for Solr to 'watch' a few files .

          say

          core.registerConfFileListener(listener, List<String> files);
          

          this would need a new interface

          public interface ConfFileListener {
             public void confFileChanged(List<String> changedFiles);
          }
          
          
          Show
          Noble Paul added a comment - - edited The possible events as I see are config file changes , commit /optimize. commit/optimize are already handled . so the main requirement is to make it possible for Solr to 'watch' a few files . say core.registerConfFileListener(listener, List< String > files); this would need a new interface public interface ConfFileListener { public void confFileChanged(List< String > changedFiles); }
          Hide
          Hoss Man added a comment -

          Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

          http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

          Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

          A unique token for finding these 240 issues in the future: hossversioncleanup20100527

          Show
          Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
          Hide
          Jan Høydahl added a comment -

          Monitoring files is not enough. It must work also for SolrCloud/ZK. A hook for coreReloaded() and one for configChanged() sounds reasonable

          Show
          Jan Høydahl added a comment - Monitoring files is not enough. It must work also for SolrCloud/ZK. A hook for coreReloaded() and one for configChanged() sounds reasonable
          Show
          Otis Gospodnetic added a comment - - edited +1 Related: http://search-lucene.com/m/2ExL22IsDPk1/solr-1307&subj=Reloading+synonyms+txt+without+downtime SOLR-2465
          Hide
          Robert Muir added a comment -

          Bulk move 3.2 -> 3.3

          Show
          Robert Muir added a comment - Bulk move 3.2 -> 3.3
          Hide
          Robert Muir added a comment -

          3.4 -> 3.5

          Show
          Robert Muir added a comment - 3.4 -> 3.5
          Hide
          Jan Høydahl added a comment -

          @SimonRosenthal, last september you said over at SOLR-2202 that you were working on this issue. Any hope for a patch which also could benefit MoneyFieldType?

          Show
          Jan Høydahl added a comment - @SimonRosenthal, last september you said over at SOLR-2202 that you were working on this issue. Any hope for a patch which also could benefit MoneyFieldType?
          Hide
          Hoss Man added a comment -

          Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently.

          email notification suppressed to prevent mass-spam
          psuedo-unique token identifying these issues: hoss20120321nofix36

          Show
          Hoss Man added a comment - Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently. email notification suppressed to prevent mass-spam psuedo-unique token identifying these issues: hoss20120321nofix36
          Hide
          Steve Rowe added a comment -

          Bulk move 4.4 issues to 4.5 and 5.0

          Show
          Steve Rowe added a comment - Bulk move 4.4 issues to 4.5 and 5.0
          Hide
          Uwe Schindler added a comment -

          Move issue to Solr 4.9.

          Show
          Uwe Schindler added a comment - Move issue to Solr 4.9.

            People

            • Assignee:
              Unassigned
              Reporter:
              Shalin Shekhar Mangar
            • Votes:
              4 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development