Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Trunk
    • Component/s: None
    • Labels:
      None

      Description

      I think for 5.0, we should make it easier to add plugins by defining a plugin package, ala a Hadoop Job jar, which is a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically.

        Issue Links

          Activity

          Hide
          Jan Høydahl added a comment -

          What happens if a plugin project uses one of the same dependent jars as Solr, but packages a wildly different version than the version we package?

          I think we don't need to re-invent OSGI to get a better plugin regime for Solr. We can document simple requirements for developers to follow.

          • Never include libraries or classes that is already part of core Lucene/Solr
          • In your solrplugin.properties, list the Solr version(s) that the plugin is tested with (and our tooling could require a --force option to disregard this and install anyway)
          • etc

          In the first version we can then simply add all jars in the plugin's /lib folder to classloader. Then if a future version of Solr causes trouble for an older plugin, the plugin maintainer must release a compatible update. When it comes to clashes between different 3rd party plugins we can tackle that with more advanced measures when it happens, or plugin developers could treat such cases as bugs and provide a fix themselves. For now let's keep it simple.

          Show
          Jan Høydahl added a comment - What happens if a plugin project uses one of the same dependent jars as Solr, but packages a wildly different version than the version we package? I think we don't need to re-invent OSGI to get a better plugin regime for Solr. We can document simple requirements for developers to follow. Never include libraries or classes that is already part of core Lucene/Solr In your solrplugin.properties , list the Solr version(s) that the plugin is tested with (and our tooling could require a --force option to disregard this and install anyway) etc In the first version we can then simply add all jars in the plugin's /lib folder to classloader. Then if a future version of Solr causes trouble for an older plugin, the plugin maintainer must release a compatible update. When it comes to clashes between different 3rd party plugins we can tackle that with more advanced measures when it happens, or plugin developers could treat such cases as bugs and provide a fix themselves. For now let's keep it simple.
          Hide
          Upayavira added a comment -

          All the above seems to be describing OSGi, for which we have the Apache Felix project here at Apache. If we are considering "plugins" that can hold conflicting dependencies, that does seem the obvious way to go.

          Show
          Upayavira added a comment - All the above seems to be describing OSGi, for which we have the Apache Felix project here at Apache. If we are considering "plugins" that can hold conflicting dependencies, that does seem the obvious way to go.
          Hide
          Erik Hatcher added a comment -

          Actually, wt=velocity&v.template.x allow to serve a velocity template around the search results.

          And, if admin-extra was loaded from classpath, there might be a way to wire it in into UI automatically. It may not be pretty on the first round, but at least something like /browse might be more possible than expected.

          VrW has a VelocitySolrResourceLoader which pulls "templates" (anything textual would work), but it is constrained to only look under a `velocity/` sub-tree from the resource loader root.

          I'm not quite getting what you're getting at here though, but one can #parse another velocity template or #include anything textual with no parsing. And easily wrap a response with a layout to get headers/footers, etc.

          Show
          Erik Hatcher added a comment - Actually, wt=velocity&v.template.x allow to serve a velocity template around the search results. And, if admin-extra was loaded from classpath, there might be a way to wire it in into UI automatically. It may not be pretty on the first round, but at least something like /browse might be more possible than expected. VrW has a VelocitySolrResourceLoader which pulls "templates" (anything textual would work), but it is constrained to only look under a `velocity/` sub-tree from the resource loader root. I'm not quite getting what you're getting at here though, but one can #parse another velocity template or #include anything textual with no parsing. And easily wrap a response with a layout to get headers/footers, etc.
          Hide
          Alexandre Rafalovitch added a comment -

          Only the classloader leakage hell! :=(

          There is no standard mechanism for inlining the dependencies, though there are a couple of different approaches, all doing weird and less than wonderful things with classloader mechanisms. Something like http://maven.apache.org/plugins/maven-shade-plugin/ and http://docs.spring.io/spring-boot/docs/1.3.0.BUILD-SNAPSHOT/reference/htmlsingle/#executable-jar

          Or, of course, OSGI http://www.javaworld.com/article/2075836/description-of-osgi-layer.html

          Show
          Alexandre Rafalovitch added a comment - Only the classloader leakage hell! :=( There is no standard mechanism for inlining the dependencies, though there are a couple of different approaches, all doing weird and less than wonderful things with classloader mechanisms. Something like http://maven.apache.org/plugins/maven-shade-plugin/ and http://docs.spring.io/spring-boot/docs/1.3.0.BUILD-SNAPSHOT/reference/htmlsingle/#executable-jar Or, of course, OSGI http://www.javaworld.com/article/2075836/description-of-osgi-layer.html
          Hide
          Shawn Heisey added a comment -

          If someone wants to offer a plugin with dependency jars , please make a jar with all the dependencies included. Once you make that assumption everything else becomes simpler.

          I worry about this idea a little bit. Maybe I don't need to be worried. What happens if a plugin project uses one of the same dependent jars as Solr, but packages a wildly different version than the version we package? Are there any situations where that might cause a larger problem than using a little more memory?

          Show
          Shawn Heisey added a comment - If someone wants to offer a plugin with dependency jars , please make a jar with all the dependencies included. Once you make that assumption everything else becomes simpler. I worry about this idea a little bit. Maybe I don't need to be worried. What happens if a plugin project uses one of the same dependent jars as Solr, but packages a wildly different version than the version we package? Are there any situations where that might cause a larger problem than using a little more memory?
          Hide
          Noble Paul added a comment -

          Well that is for us to dictate, isn't it? Who says the plugin format must conform to the JAR specs?

          It's OK for the plugin format not conform to jar specs. The payload could be a zip file with multiple jars. But, is it a big deal to merge multiple jars. It solves a problem of name collisions. What if the same FQN of a class is found in two jars , which one to load?

          and our plugin installer code handles merging all jars together into one jar plugin-xyz-merged.jar

          Please don't . The plugin installer should do as little magic as possible.

          Show
          Noble Paul added a comment - Well that is for us to dictate, isn't it? Who says the plugin format must conform to the JAR specs? It's OK for the plugin format not conform to jar specs. The payload could be a zip file with multiple jars. But, is it a big deal to merge multiple jars. It solves a problem of name collisions. What if the same FQN of a class is found in two jars , which one to load? and our plugin installer code handles merging all jars together into one jar plugin-xyz-merged.jar Please don't . The plugin installer should do as little magic as possible.
          Hide
          Jan Høydahl added a comment -

          If someone wants to offer a plugin with dependency jars , please make a jar with all the dependencies included.

          Well that is for us to dictate, isn't it? Who says the plugin format must conform to the JAR specs? Is it technically problematic to add all classes from multiple JARs from within a ZIP to a classloader? If yes, the user-facing plugin format could still be a zip with multiple jars in a lib folder, and our plugin installer code handles merging all jars together into one jar plugin-xyz-merged.jar which is the one being registered with the classloader?

          Show
          Jan Høydahl added a comment - If someone wants to offer a plugin with dependency jars , please make a jar with all the dependencies included. Well that is for us to dictate, isn't it? Who says the plugin format must conform to the JAR specs? Is it technically problematic to add all classes from multiple JARs from within a ZIP to a classloader? If yes, the user-facing plugin format could still be a zip with multiple jars in a lib folder, and our plugin installer code handles merging all jars together into one jar plugin-xyz-merged.jar which is the one being registered with the classloader?
          Hide
          Noble Paul added a comment -

          If someone wants to offer a plugin with dependency jars , please make a jar with all the dependencies included. Once you make that assumption everything else becomes simpler.

          Then it is still possible to write a script which runs as follows

          /bin/solr install-plugin xyz.jar -conf xyz.json -c collection1
          

          or we can package the xyz.jar in such a way that the META-INF contains plugin.json and the command can be simplified to

          /bin/solr install-plugin xyz.jar -c collection1
          

          Size limitation in blob store?

          as of now it is set to 5MB by default . However the user can increase it to any number with a command

          Show
          Noble Paul added a comment - If someone wants to offer a plugin with dependency jars , please make a jar with all the dependencies included. Once you make that assumption everything else becomes simpler. Then it is still possible to write a script which runs as follows /bin/solr install-plugin xyz.jar -conf xyz.json -c collection1 or we can package the xyz.jar in such a way that the META-INF contains plugin.json and the command can be simplified to /bin/solr install-plugin xyz.jar -c collection1 Size limitation in blob store? as of now it is set to 5MB by default . However the user can increase it to any number with a command
          Hide
          Alexandre Rafalovitch added a comment -

          My position is still that madness lies that way if we try to reinvent plugin management from scratch and that 3rd party solution may make better sense for that. Even if that solution is not perfect.

          But, on the other hand, I don't know internals of Solr well enough to know whether there is enough extensibility built-in to allow that.

          Show
          Alexandre Rafalovitch added a comment - My position is still that madness lies that way if we try to reinvent plugin management from scratch and that 3rd party solution may make better sense for that. Even if that solution is not perfect. But, on the other hand, I don't know internals of Solr well enough to know whether there is enough extensibility built-in to allow that.
          Hide
          Jan Høydahl added a comment - - edited

          I think that's a little more to a plugin than upload mechanics. There is a need for meta-data convention, dependency handling, versioning, etc. Supplementary UI.

          Totally agree. But we need to start in one end. Perhaps defining the minimum zip/jar layout for what to constitute a plugin. What if we choose our own file ending, e.g. .solrplugin which is really just a zip file. Then a possible layout could be

          my-plugin.solrplugin
          /solrplugin.properties -- defining e.g. name, version, dependencies etc
          /lib/  (jars to be added to classpath)
          /config/config-api-commands.json  (a series of commands to be run towards config API)
          /config/schema-api-commands.json  (a series of commands to be run towards config API)
          

          Then, over time, we can evolve the spec and add support for pluggable UI etc.

          There are tons of questions to address too

          • Size limitation in blob store?
          • Should Solr unpack all libs from the .solrplugin pkg and add them individually to blob store, or write a classloader that adds everything directly from the zip?
          • What about version clash of dependencies - should not jeopardize the rest of the system
          • Should a plugin be allowed to depend on another plugin, i.e. fail install unless the other plugin is installed?
          • Should all plugins be system-level, and then require a new config-API /collection/config "install-plugin" command to enable it for each collection?
          • What about system-level plugins, such as Authentication and Authorization plugins? Should security.json be auto updated when installing an auth plugin, or only if it does not exist already?
          • There should be a way to install plugins without registering the component with APIs, e.g. bin/solr installplugin solrcell -noregister
          • Uninstall of a plugin - it should also be able to unregister things from config / schema, for all collections where it is enabled (scary)

          First step is to see if there is enough committer interest in going down this path, then split things up into many smaller tasks that can be handled separately.

          Show
          Jan Høydahl added a comment - - edited I think that's a little more to a plugin than upload mechanics. There is a need for meta-data convention, dependency handling, versioning, etc. Supplementary UI. Totally agree. But we need to start in one end. Perhaps defining the minimum zip/jar layout for what to constitute a plugin. What if we choose our own file ending, e.g. .solrplugin which is really just a zip file. Then a possible layout could be my-plugin.solrplugin /solrplugin.properties -- defining e.g. name, version, dependencies etc /lib/ (jars to be added to classpath) /config/config-api-commands.json (a series of commands to be run towards config API) /config/schema-api-commands.json (a series of commands to be run towards config API) Then, over time, we can evolve the spec and add support for pluggable UI etc. There are tons of questions to address too Size limitation in blob store? Should Solr unpack all libs from the .solrplugin pkg and add them individually to blob store, or write a classloader that adds everything directly from the zip? What about version clash of dependencies - should not jeopardize the rest of the system Should a plugin be allowed to depend on another plugin, i.e. fail install unless the other plugin is installed? Should all plugins be system-level, and then require a new config-API /collection/config "install-plugin" command to enable it for each collection? What about system-level plugins, such as Authentication and Authorization plugins? Should security.json be auto updated when installing an auth plugin, or only if it does not exist already? There should be a way to install plugins without registering the component with APIs, e.g. bin/solr installplugin solrcell -noregister Uninstall of a plugin - it should also be able to unregister things from config / schema, for all collections where it is enabled (scary) First step is to see if there is enough committer interest in going down this path, then split things up into many smaller tasks that can be handled separately.
          Hide
          Alexandre Rafalovitch added a comment -

          Actually, wt=velocity&v.template.x allow to serve a velocity template around the search results. Unless broken by recent changes, etc. Erik Hatcher may be able to add details.

          And, if admin-extra was loaded from classpath, there might be a way to wire it in into UI automatically.

          It may not be pretty on the first round, but at least something like /browse might be more possible than expected.

          Show
          Alexandre Rafalovitch added a comment - Actually, wt=velocity&v.template.x allow to serve a velocity template around the search results. Unless broken by recent changes, etc. Erik Hatcher may be able to add details. And, if admin-extra was loaded from classpath, there might be a way to wire it in into UI automatically. It may not be pretty on the first round, but at least something like /browse might be more possible than expected.
          Hide
          Jan Høydahl added a comment -

          So you are trying to eliminate step #3 here. Is that right ?

          Not primarily. Your list is over simplified. Today:

          1. Realize that there is a plugin for what you want to do
          2. Locate and download that plugin
          3. Read the docs and find and download any dependencies of the plugin
            • Example: dist/solr-cell-5.0.0.jar is the plugin, and you also have 34 dependency jars from contrib/lib/
          4. For each jar required (35 in total):
          5. For each collection in the system (that needs the plugin):
            • For each jar that belongs to the "plugin":
              • Put each jar on the "classpath" for the collection
              • curl http://localhost:8983/solr/collection1/config -H 'Content-type:application/json'  -d '{
                "add-runtimelib" : {"name": "jarname" , "version":2 },
                "update-runtimelib" :{"name": "jarname" ,"version":3},
                "delete-runtimelib" :"jarname" 
                }' 
                
              • Register the plugin into config or schema or zookeeper or solr.xml depending on the type, e.g.
              • {"add-searchcomponent":
                   "name":"elevator",
                   "class":"QueryElevationComponent",
                   "queryFieldType":"string",
                   "config-file":"elevate.xml"
                }
                

          Not to mention when you want to upgrade the plugin to a newer version, or uninstall it..

          Now compare this to a click in the Admin UI or:

          bin/solr installplugin solrcell 5.2.1
          bin/solr removeplugin solrcell
          bin/solr installplugin solrcell 5.3.0
          
          Show
          Jan Høydahl added a comment - So you are trying to eliminate step #3 here. Is that right ? Not primarily. Your list is over simplified. Today: Realize that there is a plugin for what you want to do Locate and download that plugin Read the docs and find and download any dependencies of the plugin Example: dist/solr-cell-5.0.0.jar is the plugin, and you also have 34 dependency jars from contrib/lib/ For each jar required (35 in total): {{curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @filename.jar http://localhost:8983/solr/.system/blob/name }} For each collection in the system (that needs the plugin): For each jar that belongs to the "plugin": Put each jar on the "classpath" for the collection curl http: //localhost:8983/solr/collection1/config -H 'Content-type:application/json' -d '{ "add-runtimelib" : { "name" : "jarname" , "version" :2 }, "update-runtimelib" :{ "name" : "jarname" , "version" :3}, "delete-runtimelib" : "jarname" }' Register the plugin into config or schema or zookeeper or solr.xml depending on the type, e.g. { "add-searchcomponent" : "name" : "elevator" , "class" : "QueryElevationComponent" , "queryFieldType" : "string" , "config-file" : "elevate.xml" } Not to mention when you want to upgrade the plugin to a newer version, or uninstall it.. Now compare this to a click in the Admin UI or: bin/solr installplugin solrcell 5.2.1 bin/solr removeplugin solrcell bin/solr installplugin solrcell 5.3.0
          Hide
          Noble Paul added a comment -

          We don't yet have a concept of a plugin offering a UI .

          Show
          Noble Paul added a comment - We don't yet have a concept of a plugin offering a UI .
          Hide
          Alexandre Rafalovitch added a comment -

          I think that's a little more to a plugin than upload mechanics. There is a need for meta-data convention, dependency handling, versioning, etc. Supplementary UI.

          So, making upload mechanics could be a good first step, but the bigger picture needs to be kept in mind as well.

          BTW, Velocity loader apparently can load pages from the classpath. So, it might be possible to bundle pages with the plugin somehow already, if the classpath is setup right.

          Show
          Alexandre Rafalovitch added a comment - I think that's a little more to a plugin than upload mechanics. There is a need for meta-data convention, dependency handling, versioning, etc. Supplementary UI. So, making upload mechanics could be a good first step, but the bigger picture needs to be kept in mind as well. BTW, Velocity loader apparently can load pages from the classpath. So, it might be possible to bundle pages with the plugin somehow already, if the classpath is setup right.
          Hide
          Noble Paul added a comment - - edited

          I guess I'm getting what you are trying to say

          Now we need to follow the following steps to register a plugin from a jar (using SOLR-7073)

          1. upload the jar to .system collection
          2. add the jar to collection classpath using the add-runtime-lib command
          3. add the component using the add-<component-name> command

          So you are trying to eliminate step #3 here. Is that right ?

          Show
          Noble Paul added a comment - - edited I guess I'm getting what you are trying to say Now we need to follow the following steps to register a plugin from a jar (using SOLR-7073 ) upload the jar to .system collection add the jar to collection classpath using the add-runtime-lib command add the component using the add-<component-name> command So you are trying to eliminate step #3 here. Is that right ?
          Hide
          Jan Høydahl added a comment -

          The issue description is quite describing, isn't it? We're on-topic here as far as I can tell:

          I think for 5.0, we should make it easier to add plugins by defining a plugin package, ala a Hadoop Job jar, which is a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically.

          Solr's "plugins" today are really just Java classes that happen to implement a certain interface that we have defined as a plugin, and then it is up to the user to get hold of the plugin from <somewhere>, find a way to place it on the classpath, register the full class name into appropriate config file (varies depending on what the plugin does), restart Solr and then start using it.

          This JIRA is trying to define a broader pluging definition, where the Java class is just part of it all, where dependency jars and configuration could be packaged with the plugin, where the whole fetch-install-configure complexity is hidden and can be done by the click of a GUI button or running of one shell command.

          Besides - how many Solr "plugins" do you know of today in "the wild"? How do you find them? Most are just patches in JIRA, others have their own installers, yet others include some description of how to copy thing into various places, editing some XML files etc. The 3rd party "market" for cool new features will probably take off once we can offer such a simplified plugin architecture. And we won't force anyone, you can still plug classes in the manual way if you like.

          Show
          Jan Høydahl added a comment - The issue description is quite describing, isn't it? We're on-topic here as far as I can tell: I think for 5.0, we should make it easier to add plugins by defining a plugin package, ala a Hadoop Job jar, which is a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically. Solr's "plugins" today are really just Java classes that happen to implement a certain interface that we have defined as a plugin, and then it is up to the user to get hold of the plugin from <somewhere>, find a way to place it on the classpath, register the full class name into appropriate config file (varies depending on what the plugin does), restart Solr and then start using it. This JIRA is trying to define a broader pluging definition, where the Java class is just part of it all, where dependency jars and configuration could be packaged with the plugin, where the whole fetch-install-configure complexity is hidden and can be done by the click of a GUI button or running of one shell command. Besides - how many Solr "plugins" do you know of today in "the wild"? How do you find them? Most are just patches in JIRA, others have their own installers, yet others include some description of how to copy thing into various places, editing some XML files etc. The 3rd party "market" for cool new features will probably take off once we can offer such a simplified plugin architecture. And we won't force anyone, you can still plug classes in the manual way if you like.
          Hide
          Noble Paul added a comment -

          I fail to see what are you trying to solve here. Please give the problem statement and state the proposed solution. I guess it is worth opening a new ticket

          Show
          Noble Paul added a comment - I fail to see what are you trying to solve here. Please give the problem statement and state the proposed solution. I guess it is worth opening a new ticket
          Hide
          Jan Høydahl added a comment - - edited

          After we got SOLR-7073, we can now add a single .jar to a collection by uploading it to .system collection. Could we perhaps piggyback on this mechanism for full-blown plugins as well?

          I.e. we similarly upload a plugin, e.g. my-cool-responsewriter-plugin.zip to .system. Solr will then automatically inspect the zip, add jars to classpath and register the new responsewriter under its default name.

          On node start, we would iterate over all registered plugins in .system and register them. Perhaps a super generic way to support auto wiring is to allow scripts in the zip root, like config-api-commands.json and schema-api-commands.json, which will be replayed on load. Then a FieldType plugin can e.g. register itself with a fieldType and dynamicField in schema.

          The whole idea of such a zip format would be to encourage creation of 3rd party plugins that can be installed from AdminUI or bin/solr installplugin my-plugin.zip, and perhaps also more modularization of the Solr distro, i.e. our contribs would become plugins and not litter solrconfig.xml with config that is only active if a system property is enabled.

          How cool would it be if you could add e.g. SolrCell to a plain Solr download by typing bin/solr installplugin solrcell, and it would look for http://www.us.apache.org/dist/lucene/solr/_version_/plugins/solrcell.zip. Could also support github sources, i.e. bin/solr installplugin githubuser/plugin-name where plugin-name is a repo with some simple layout rules.

          To aid in user friendliness, solrconfig.xml could even have a new tag to to register expected dependencies:

          <dependencies>
             <plugin name="solrcell" version="5.2.1" />
             <plugin name="cominvent/cool-responsewriter" version=">=5.0.0" />
          </dependencies>
          

          ...and spit out clear instructions in the logs on how to install those.

          Show
          Jan Høydahl added a comment - - edited After we got SOLR-7073 , we can now add a single .jar to a collection by uploading it to .system collection. Could we perhaps piggyback on this mechanism for full-blown plugins as well? I.e. we similarly upload a plugin, e.g. my-cool-responsewriter-plugin.zip to .system . Solr will then automatically inspect the zip, add jars to classpath and register the new responsewriter under its default name. On node start, we would iterate over all registered plugins in .system and register them. Perhaps a super generic way to support auto wiring is to allow scripts in the zip root, like config-api-commands.json and schema-api-commands.json , which will be replayed on load. Then a FieldType plugin can e.g. register itself with a fieldType and dynamicField in schema. The whole idea of such a zip format would be to encourage creation of 3rd party plugins that can be installed from AdminUI or bin/solr installplugin my-plugin.zip , and perhaps also more modularization of the Solr distro, i.e. our contribs would become plugins and not litter solrconfig.xml with config that is only active if a system property is enabled. How cool would it be if you could add e.g. SolrCell to a plain Solr download by typing bin/solr installplugin solrcell , and it would look for http://www.us.apache.org/dist/lucene/solr/_version_/plugins/solrcell.zip . Could also support github sources, i.e. bin/solr installplugin githubuser/plugin-name where plugin-name is a repo with some simple layout rules. To aid in user friendliness, solrconfig.xml could even have a new tag to to register expected dependencies: <dependencies> <plugin name= "solrcell" version= "5.2.1" /> <plugin name= "cominvent/cool-responsewriter" version= "> =5.0.0" /> </dependencies> ...and spit out clear instructions in the logs on how to install those.
          Jan Høydahl made changes -
          Link This issue is related to SOLR-7073 [ SOLR-7073 ]
          Shawn Heisey made changes -
          Link This issue relates to SOLR-6806 [ SOLR-6806 ]
          Hide
          Alexandre Rafalovitch added a comment -

          What about something like OSGI? It might be a little heavier but I suspect addresses a bunch of questions that we would be fighting through until Solr 7 otherwise.

          And there is a corresponding Apache project: http://felix.apache.org/ (and others on top, I think). Which, I am sure, comes with built-in community and tooling support.

          Show
          Alexandre Rafalovitch added a comment - What about something like OSGI? It might be a little heavier but I suspect addresses a bunch of questions that we would be fighting through until Solr 7 otherwise. And there is a corresponding Apache project: http://felix.apache.org/ (and others on top, I think). Which, I am sure, comes with built-in community and tooling support.
          Hide
          Jan Høydahl added a comment -

          There are existing lightweight frameworks out there, such as https://github.com/decebals/pf4j, loading all plugins found in a folder. Looks pretty extensible..

          Adding a bin/plugin script for install/uninstall from a repo or github would also be cool.

          Show
          Jan Høydahl added a comment - There are existing lightweight frameworks out there, such as https://github.com/decebals/pf4j , loading all plugins found in a folder. Looks pretty extensible.. Adding a bin/plugin script for install/uninstall from a repo or github would also be cool.
          Hide
          Alexandre Rafalovitch added a comment -

          We don't yet have a uniform way to manage code+UI . To be honest , all UI is kind of an afterthought in Solr

          Well, it will continue being an afterthought if you continue not thinking about it.

          May I suggest a thought experiment? Let's imagine we are shipping Solr Javadoc as a plugin. And it will be searchable (e.g. come with pre-built Solr collection). And the HTML itself is - ideally - compressed and is served that way. Now, we can walk through the aspects required to make it work. Maybe they will not all make it into 5, but the pieces that do line up should be on the critical path of making a specific scenario happen.

          The problem with plugins is that they seem easy but the real-life consequences of them get hairy very fast. I think Solr desperately needs plugins, but they need to be a bit more than a class in a jar. There needs to be some sort of management/flow/metadata to avoid fragmentation and user confusion. Does not have to be super-comprehensive in initial setup, just with some advance forethought.

          Show
          Alexandre Rafalovitch added a comment - We don't yet have a uniform way to manage code+UI . To be honest , all UI is kind of an afterthought in Solr Well, it will continue being an afterthought if you continue not thinking about it. May I suggest a thought experiment? Let's imagine we are shipping Solr Javadoc as a plugin. And it will be searchable (e.g. come with pre-built Solr collection). And the HTML itself is - ideally - compressed and is served that way. Now, we can walk through the aspects required to make it work. Maybe they will not all make it into 5, but the pieces that do line up should be on the critical path of making a specific scenario happen. The problem with plugins is that they seem easy but the real-life consequences of them get hairy very fast. I think Solr desperately needs plugins, but they need to be a bit more than a class in a jar. There needs to be some sort of management/flow/metadata to avoid fragmentation and user confusion. Does not have to be super-comprehensive in initial setup, just with some advance forethought.
          Hide
          Noble Paul added a comment -

          We don't yet have a uniform way to manage code+UI . To be honest , all UI is kind of an afterthought in Solr

          Show
          Noble Paul added a comment - We don't yet have a uniform way to manage code+UI . To be honest , all UI is kind of an afterthought in Solr
          Hide
          Alexandre Rafalovitch added a comment -

          It may be worth considering that plugin may contain more than code. For example, it may contain AdminUI screens, similar to those admin-extra files. Elasticsearch has something like this.

          There might also be some sort of dependency in there to avoid jar rabbits (multiple copies of the same shaded jars). Elasticsarch does NOT seem to have something like this.

          On the positive side, a website to track the plugins and dependencies would make for a very nice Solr demo project and would help the community actually start contributing things as plugins, once the visibility part is solved

          Show
          Alexandre Rafalovitch added a comment - It may be worth considering that plugin may contain more than code. For example, it may contain AdminUI screens, similar to those admin-extra files. Elasticsearch has something like this. There might also be some sort of dependency in there to avoid jar rabbits (multiple copies of the same shaded jars). Elasticsarch does NOT seem to have something like this. On the positive side, a website to track the plugins and dependencies would make for a very nice Solr demo project and would help the community actually start contributing things as plugins, once the visibility part is solved
          Noble Paul made changes -
          Link This issue is related to SOLR-6607 [ SOLR-6607 ]
          Hide
          Jan Høydahl added a comment -

          Also see from mailing list: http://search-lucene.com/m/WwzTb2jQWpl1

          Show
          Jan Høydahl added a comment - Also see from mailing list: http://search-lucene.com/m/WwzTb2jQWpl1
          Hide
          Mark Miller added a comment -

          I think we would still need a configuration option that does not involve Java. Other than that, I don't really see too many issues. As far as names, it could possibly default to the name of the plugin jar or something else and first one wins, second one fails. We could also namespace, kind of how you see with eclipse. Honestly though, nothing sounds much better than a simple config file with the jar, same name or whatever, that has the plugin name or url. Probably that's the default minimal setup - even if you can delete that file and run with some default names or whatever for plugins that don't require any config.

          I think the path is obviously right - you see it used plenty elsewhere, and it resonates with me. But I think Hoss is right in that the common case will still need configuration. I don't see how the two points argue with each other though.

          We will need to be able to configure with a user friendly format - most plugins will have that - if not just to show the user what can be configured. There is no reason we have to require it though. Some plugin might simply be there to insert some custom code on startup.

          Show
          Mark Miller added a comment - I think we would still need a configuration option that does not involve Java. Other than that, I don't really see too many issues. As far as names, it could possibly default to the name of the plugin jar or something else and first one wins, second one fails. We could also namespace, kind of how you see with eclipse. Honestly though, nothing sounds much better than a simple config file with the jar, same name or whatever, that has the plugin name or url. Probably that's the default minimal setup - even if you can delete that file and run with some default names or whatever for plugins that don't require any config. I think the path is obviously right - you see it used plenty elsewhere, and it resonates with me. But I think Hoss is right in that the common case will still need configuration. I don't see how the two points argue with each other though. We will need to be able to configure with a user friendly format - most plugins will have that - if not just to show the user what can be configured. There is no reason we have to require it though. Some plugin might simply be there to insert some custom code on startup.
          Hide
          Grant Ingersoll added a comment -

          Hoss,

          A few of my thoughts on it:

          1. I don't think the config necessarily needs to be in the plugin. Part of the goal here is to be able to easily add a plugin and then easily configure it and manage it (from the UI?).
          2. Plugins should be self-contained and (ideally) class loader isolated
          3. You shouldn't necessarily have to restart Solr in order to add a plugin
          Show
          Grant Ingersoll added a comment - Hoss, A few of my thoughts on it: I don't think the config necessarily needs to be in the plugin. Part of the goal here is to be able to easily add a plugin and then easily configure it and manage it (from the UI?). Plugins should be self-contained and (ideally) class loader isolated You shouldn't necessarily have to restart Solr in order to add a plugin
          Hide
          Hoss Man added a comment -

          Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath.

          Ok, so something like a QParserPlugin could optionally register itself using a "default" name hardcoded in it's code (similar to how SolrCore auto-wires the "default" QParsers that come with solr-core out of hte box) ... but how would that work if multiple plugins try to use the same name? (in solr-core we can ensure that doesn't happen, and we don't register something with the same name as a qparser explicitly registered via users configuration) Not to mention you'd still need to support a method of explicitly registering QParserPlugins since it might support init params that you want to specify (none of hte out of the box QParsers do, but it's possible, and we shouldn't take that customization away)

          QParsers really seem like the "special case" not the common case as far as plugins go and this sort of hypothetical "automatic loading & registration" ... FieldTypes, Analysis factories, search components, request handlers, ... these are all things i'm hard pressed to imagine being useul w/o some sort of explicit "this is how i want to use it" type information from the person admining the solr instance.

          Show
          Hoss Man added a comment - Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath. Ok, so something like a QParserPlugin could optionally register itself using a "default" name hardcoded in it's code (similar to how SolrCore auto-wires the "default" QParsers that come with solr-core out of hte box) ... but how would that work if multiple plugins try to use the same name? (in solr-core we can ensure that doesn't happen, and we don't register something with the same name as a qparser explicitly registered via users configuration) Not to mention you'd still need to support a method of explicitly registering QParserPlugins since it might support init params that you want to specify (none of hte out of the box QParsers do, but it's possible, and we shouldn't take that customization away) QParsers really seem like the "special case" not the common case as far as plugins go and this sort of hypothetical "automatic loading & registration" ... FieldTypes, Analysis factories, search components, request handlers, ... these are all things i'm hard pressed to imagine being useul w/o some sort of explicit "this is how i want to use it" type information from the person admining the solr instance.
          Hide
          Yonik Seeley added a comment -

          can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance?

          Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath.

          are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar?

          I hope not. That would seem like a step backwards in usability.

          Show
          Yonik Seeley added a comment - can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance? Small stuff shouldn't need any configuration... for instance a custom QParser should just be able to work if it's in the classpath. are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar? I hope not. That would seem like a step backwards in usability.
          Hide
          Hoss Man added a comment -

          a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically.

          +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up".

          can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance?

          For all the examples i can think of, this doens't really make sens to me – specifically: all of the existing plugins you can find in "solr/contrib" ... are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar? are we expecting people who want to use the langid update processor to re-jar it with some bit of config snippet that tells it what fields to populate and what langauge to look for? ... and then what? they drop that jar into some directory, and it still does nothing until they programatically use some API to say which of their (possibly may) update processor chains they want it to be a part of, and where in the chain it should execute?

          Show
          Hoss Man added a comment - a self--contained archive of a plugin that can be easily installed (even from the UI!) and configured programmatically. +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up". can someone give me a practical/hypothetical example of what this would look like to someone setting up a solr instance? For all the examples i can think of, this doens't really make sens to me – specifically: all of the existing plugins you can find in "solr/contrib" ... are we expecting users to JAR up their own "copy" of DIH with it's config inside of it's jar? are we expecting people who want to use the langid update processor to re-jar it with some bit of config snippet that tells it what fields to populate and what langauge to look for? ... and then what? they drop that jar into some directory, and it still does nothing until they programatically use some API to say which of their (possibly may) update processor chains they want it to be a part of, and where in the chain it should execute?
          Hide
          Uwe Schindler added a comment -

          Theoretically you can put references to additional JARs, that are needed for running, into the META-INF/MANIFEST.MF file as classpath. I have not tried this, but it works e.g. for java -jar something.jar. If something.jar has additional JARs in its manifest, those are added to classpath. See http://docs.oracle.com/javase/tutorial/deployment/jar/downman.html

          Show
          Uwe Schindler added a comment - Theoretically you can put references to additional JARs, that are needed for running, into the META-INF/MANIFEST.MF file as classpath. I have not tried this, but it works e.g. for java -jar something.jar . If something.jar has additional JARs in its manifest, those are added to classpath. See http://docs.oracle.com/javase/tutorial/deployment/jar/downman.html
          Hide
          Grant Ingersoll added a comment -

          If the plugin has dependency jars, do you expect it to be a part of the jar itself?

          Yes, I'm thinking something along the lines of Hadoop Job Jar.

          Show
          Grant Ingersoll added a comment - If the plugin has dependency jars, do you expect it to be a part of the jar itself? Yes, I'm thinking something along the lines of Hadoop Job Jar.
          Hide
          Noble Paul added a comment -

          In any plugin loading , we may just need a Plugin class name and probably that is it

          If the plugin has dependency jars, do you expect it to be a part of the jar itself?

          Show
          Noble Paul added a comment - In any plugin loading , we may just need a Plugin class name and probably that is it If the plugin has dependency jars, do you expect it to be a part of the jar itself?
          Hide
          Grant Ingersoll added a comment -

          Tying this in w/ Guice, I think we can get away w/o any meta-data, but instead the plugin will need to provide an AbstractModule (maybe even not) and then they can configure the plugin via an API, or it can also default.

          Show
          Grant Ingersoll added a comment - Tying this in w/ Guice, I think we can get away w/o any meta-data, but instead the plugin will need to provide an AbstractModule (maybe even not) and then they can configure the plugin via an API, or it can also default.
          Hide
          Noble Paul added a comment -

          does it have a custom format for the plugin jar? some kind of a meta-data along with the jar?

          Show
          Noble Paul added a comment - does it have a custom format for the plugin jar? some kind of a meta-data along with the jar?
          Grant Ingersoll made changes -
          Field Original Value New Value
          Link This issue Is contained by SOLR-4818 [ SOLR-4818 ]
          Hide
          Grant Ingersoll added a comment - - edited

          I don't see any downside to it, but the classloader stuff can get real hairy.

          It could be for 4, but I don't want to worry about backporting just yet.

          Show
          Grant Ingersoll added a comment - - edited I don't see any downside to it, but the classloader stuff can get real hairy. It could be for 4, but I don't want to worry about backporting just yet.
          Hide
          Yonik Seeley added a comment -

          +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up".

          Should be noted that we would want classloader isolation for this packaging.

          Any downsides to that?
          Also, any reason it can't be in 4x (provided it's done in time?)

          Show
          Yonik Seeley added a comment - +1 for this! Keep the configuration w/ the plugin and don't require any central place to "wire it up". Should be noted that we would want classloader isolation for this packaging. Any downsides to that? Also, any reason it can't be in 4x (provided it's done in time?)
          Hide
          Grant Ingersoll added a comment -

          Should be noted that we would want classloader isolation for this packaging.

          Show
          Grant Ingersoll added a comment - Should be noted that we would want classloader isolation for this packaging.
          Hide
          Grant Ingersoll added a comment -

          https://code.google.com/p/google-guice/wiki/Multibindings has some baseline good ideas in it, see SOLR-5091 as well for how Guice gets brought in.

          Show
          Grant Ingersoll added a comment - https://code.google.com/p/google-guice/wiki/Multibindings has some baseline good ideas in it, see SOLR-5091 as well for how Guice gets brought in.
          Grant Ingersoll created issue -

            People

            • Assignee:
              Grant Ingersoll
              Reporter:
              Grant Ingersoll
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:

                Development