Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 5.0
    • Component/s: None
    • Labels:
      None

      Description

      I think for 5.0, we should make it easier to add plugins by defining a plugin package, ala a Hadoop Job jar, which is a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically.

        Issue Links

          Activity

          Hide
          Grant Ingersoll added a comment -

          https://code.google.com/p/google-guice/wiki/Multibindings has some baseline good ideas in it, see SOLR-5091 as well for how Guice gets brought in.

          Show
          Grant Ingersoll added a comment - https://code.google.com/p/google-guice/wiki/Multibindings has some baseline good ideas in it, see SOLR-5091 as well for how Guice gets brought in.
          Hide
          Grant Ingersoll added a comment -

          Should be noted that we would want classloader isolation for this packaging.

          Show
          Grant Ingersoll added a comment - Should be noted that we would want classloader isolation for this packaging.
          Hide
          Yonik Seeley added a comment -

          +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up".

          Should be noted that we would want classloader isolation for this packaging.

          Any downsides to that?
          Also, any reason it can't be in 4x (provided it's done in time?)

          Show
          Yonik Seeley added a comment - +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up". Should be noted that we would want classloader isolation for this packaging. Any downsides to that? Also, any reason it can't be in 4x (provided it's done in time?)
          Hide
          Grant Ingersoll added a comment - - edited

          I don't see any downside to it, but the classloader stuff can get real hairy.

          It could be for 4, but I don't want to worry about backporting just yet.

          Show
          Grant Ingersoll added a comment - - edited I don't see any downside to it, but the classloader stuff can get real hairy. It could be for 4, but I don't want to worry about backporting just yet.
          Hide
          Noble Paul added a comment -

          does it have a custom format for the plugin jar? some kind of a meta-data along with the jar?

          Show
          Noble Paul added a comment - does it have a custom format for the plugin jar? some kind of a meta-data along with the jar?
          Hide
          Grant Ingersoll added a comment -

          Tying this in w/ Guice, I think we can get away w/o any meta-data, but instead the plugin will need to provide an AbstractModule (maybe even not) and then they can configure the plugin via an API, or it can also default.

          Show
          Grant Ingersoll added a comment - Tying this in w/ Guice, I think we can get away w/o any meta-data, but instead the plugin will need to provide an AbstractModule (maybe even not) and then they can configure the plugin via an API, or it can also default.
          Hide
          Noble Paul added a comment -

          In any plugin loading , we may just need a Plugin class name and probably that is it

          If the plugin has dependency jars, do you expect it to be a part of the jar itself?

          Show
          Noble Paul added a comment - In any plugin loading , we may just need a Plugin class name and probably that is it If the plugin has dependency jars, do you expect it to be a part of the jar itself?
          Hide
          Grant Ingersoll added a comment -

          If the plugin has dependency jars, do you expect it to be a part of the jar itself?

          Yes, I'm thinking something along the lines of Hadoop Job Jar.

          Show
          Grant Ingersoll added a comment - If the plugin has dependency jars, do you expect it to be a part of the jar itself? Yes, I'm thinking something along the lines of Hadoop Job Jar.
          Hide
          Uwe Schindler added a comment -

          Theoretically you can put references to additional JARs, that are needed for running, into the META-INF/MANIFEST.MF file as classpath. I have not tried this, but it works e.g. for java -jar something.jar. If something.jar has additional JARs in its manifest, those are added to classpath. See http://docs.oracle.com/javase/tutorial/deployment/jar/downman.html

          Show
          Uwe Schindler added a comment - Theoretically you can put references to additional JARs, that are needed for running, into the META-INF/MANIFEST.MF file as classpath. I have not tried this, but it works e.g. for java -jar something.jar . If something.jar has additional JARs in its manifest, those are added to classpath. See http://docs.oracle.com/javase/tutorial/deployment/jar/downman.html
          Hide
          Hoss Man added a comment -

          a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically.

          +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up".

          can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance?

          For all the examples i can think of, this doens't really make sens to me – specifically: all of the existing plugins you can find in "solr/contrib" ... are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar? are we expecting people who want to use the langid update processor to re-jar it with some bit of config snippet that tells it what fields to populate and what langauge to look for? ... and then what? they drop that jar into some directory, and it still does nothing until they programatically use some API to say which of their (possibly may) update processor chains they want it to be a part of, and where in the chain it should execute?

          Show
          Hoss Man added a comment - a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically. +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up". can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance? For all the examples i can think of, this doens't really make sens to me – specifically: all of the existing plugins you can find in "solr/contrib" ... are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar? are we expecting people who want to use the langid update processor to re-jar it with some bit of config snippet that tells it what fields to populate and what langauge to look for? ... and then what? they drop that jar into some directory, and it still does nothing until they programatically use some API to say which of their (possibly may) update processor chains they want it to be a part of, and where in the chain it should execute?
          Hide
          Yonik Seeley added a comment -

          can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance?

          Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath.

          are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar?

          I hope not. That would seem like a step backwards in usability.

          Show
          Yonik Seeley added a comment - can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance? Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath. are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar? I hope not. That would seem like a step backwards in usability.
          Hide
          Hoss Man added a comment -

          Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath.

          Ok, so something like a QParserPlugin could optionally register itself using a "default" name hardcoded in it's code (similar to how SolrCore auto-wires the "default" QParsers that come with solr-core out of hte box) ... but how would that work if multiple plugins try to use the same name? (in solr-core we can ensure that doesn't happen, and we don't register something with the same name as a qparser explicitly registered via users configuration) Not to mention you'd still need to support a method of explicitly registering QParserPlugins since it might support init params that you want to specify (none of hte out of the box QParsers do, but it's possible, and we shouldn't take that customization away)

          QParsers really seem like the "special case" not the common case as far as plugins go and this sort of hypothetical "automatic loading & registration" ... FieldTypes, Analysis factories, search components, request handlers, ... these are all things i'm hard pressed to imagine being useul w/o some sort of explicit "this is how i want to use it" type information from the person admining the solr instance.

          Show
          Hoss Man added a comment - Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath. Ok, so something like a QParserPlugin could optionally register itself using a "default" name hardcoded in it's code (similar to how SolrCore auto-wires the "default" QParsers that come with solr-core out of hte box) ... but how would that work if multiple plugins try to use the same name? (in solr-core we can ensure that doesn't happen, and we don't register something with the same name as a qparser explicitly registered via users configuration) Not to mention you'd still need to support a method of explicitly registering QParserPlugins since it might support init params that you want to specify (none of hte out of the box QParsers do, but it's possible, and we shouldn't take that customization away) QParsers really seem like the "special case" not the common case as far as plugins go and this sort of hypothetical "automatic loading & registration" ... FieldTypes, Analysis factories, search components, request handlers, ... these are all things i'm hard pressed to imagine being useul w/o some sort of explicit "this is how i want to use it" type information from the person admining the solr instance.
          Hide
          Grant Ingersoll added a comment -

          Hoss,

          A few of my thoughts on it:

          1. I don't think the config necessarily needs to be in the plugin. Part of the goal here is to be able to easily add a plugin and then easily configure it and manage it (from the UI?).
          2. Plugins should be self-contained and (ideally) class loader isolated
          3. You shouldn't necessarily have to restart Solr in order to add a plugin
          Show
          Grant Ingersoll added a comment - Hoss, A few of my thoughts on it: I don't think the config necessarily needs to be in the plugin. Part of the goal here is to be able to easily add a plugin and then easily configure it and manage it (from the UI?). Plugins should be self-contained and (ideally) class loader isolated You shouldn't necessarily have to restart Solr in order to add a plugin
          Hide
          Mark Miller added a comment -

          I think we would still need a configuration option that does not involve Java. Other than that, I don't really see too many issues. As far as names, it could possibly default to the name of the plugin jar or something else and first one wins, second one fails. We could also namespace, kind of how you see with eclipse. Honestly though, nothing sounds much better than a simple config file with the jar, same name or whatever, that has the plugin name or url. Probably that's the default minimal setup - even if you can delete that file and run with some default names or whatever for plugins that don't require any config.

          I think the path is obviously right - you see it used plenty elsewhere, and it resonates with me. But I think Hoss is right in that the common case will still need configuration. I don't see how the two points argue with each other though.

          We will need to be able to configure with a user friendly format - most plugins will have that - if not just to show the user what can be configured. There is no reason we have to require it though. Some plugin might simply be there to insert some custom code on startup.

          Show
          Mark Miller added a comment - I think we would still need a configuration option that does not involve Java. Other than that, I don't really see too many issues. As far as names, it could possibly default to the name of the plugin jar or something else and first one wins, second one fails. We could also namespace, kind of how you see with eclipse. Honestly though, nothing sounds much better than a simple config file with the jar, same name or whatever, that has the plugin name or url. Probably that's the default minimal setup - even if you can delete that file and run with some default names or whatever for plugins that don't require any config. I think the path is obviously right - you see it used plenty elsewhere, and it resonates with me. But I think Hoss is right in that the common case will still need configuration. I don't see how the two points argue with each other though. We will need to be able to configure with a user friendly format - most plugins will have that - if not just to show the user what can be configured. There is no reason we have to require it though. Some plugin might simply be there to insert some custom code on startup.

            People

            • Assignee:
              Grant Ingersoll
              Reporter:
              Grant Ingersoll
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development