Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10087

StreamHandler should be able to use runtimeLib jars

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      StreamHandler currently can't uses jars that via the runtimeLib and Blob Store api. This is because the StreamHandler uses core.getResourceLoader() instead of core.getMemClassLoader() for loading classes.

      An example of this working with the fix is here: https://github.com/risdenk/solr_custom_streaming_expressions

      Steps:

      # Inspired by https://cwiki.apache.org/confluence/display/solr/Adding+Custom+Plugins+in+SolrCloud+Mode
      
      # Start Solr with enabling Blob Store
      ./bin/solr start -c -f -a "-Denable.runtime.lib=true"
      
      # Create test collection
      ./bin/solr create -c test
      
      # Create .system collection
      curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=.system'
      
      # Build custom streaming expression jar
      (cd custom-streaming-expression && mvn clean package)
      
      # Upload jar to .system collection using Blob Store API (https://cwiki.apache.org/confluence/display/solr/Blob+Store+API)
      curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @custom-streaming-expression/target/custom-streaming-expression-1.0-SNAPSHOT.jar 'http://localhost:8983/solr/.system/blob/test'
      
      # List all blobs that are stored
      curl 'http://localhost:8983/solr/.system/blob?omitHeader=true'
      
      # Add the jar to the runtime lib
      curl 'http://localhost:8983/solr/test/config' -H 'Content-type:application/json' -d '{
         "add-runtimelib": { "name":"test", "version":1 }
      }'
      
      # Create custom streaming expression using work from SOLR-9103
      # Patch from SOLR-10087 is required for StreamHandler to load the runtimeLib jar
      curl 'http://localhost:8983/solr/test/config' -H 'Content-type:application/json' -d '{
        "create-expressible": {
          "name": "customstreamingexpression",
          "class": "com.test.solr.CustomStreamingExpression",
          "runtimeLib": true
        }
      }'
      
      # Test the custom streaming expression
      curl 'http://localhost:8983/solr/test/stream?expr=customstreamingexpression()'
      

        Issue Links

          Activity

          Hide
          risdenk Kevin Risden added a comment -
          Show
          risdenk Kevin Risden added a comment - Noble Paul Cao Manh Dat Dennis Gove Joel Bernstein - Thoughts on this?
          Hide
          dpgove Dennis Gove added a comment -

          I'm hesitant about the ability to post binaries into Solr - I feel like it's a huge security concern. That said, this feature isn't adding the ability to post binaries, just the ability to make use of posted binaries in streaming. If the ability to post binaries is something that is already considered safe and/or the benefits outweigh the drawbacks, then I don't see any reason why streaming shouldn't support it.

          Show
          dpgove Dennis Gove added a comment - I'm hesitant about the ability to post binaries into Solr - I feel like it's a huge security concern. That said, this feature isn't adding the ability to post binaries, just the ability to make use of posted binaries in streaming. If the ability to post binaries is something that is already considered safe and/or the benefits outweigh the drawbacks, then I don't see any reason why streaming shouldn't support it.
          Hide
          risdenk Kevin Risden added a comment -

          The ability to add binaries is protected by a system property and has to be explicitly enabled. Additionally, the jars can be signed if required. https://cwiki.apache.org/confluence/display/solr/Adding+Custom+Plugins+in+SolrCloud+Mode

          Show
          risdenk Kevin Risden added a comment - The ability to add binaries is protected by a system property and has to be explicitly enabled. Additionally, the jars can be signed if required. https://cwiki.apache.org/confluence/display/solr/Adding+Custom+Plugins+in+SolrCloud+Mode
          Hide
          noble.paul Noble Paul added a comment - - edited

          I believe all components should be dynamically loadable. Kevin Risden , please explain (in description) how users can register a streaming expression plugin and use it if the class is loaded from a runtime jar

          Dennis Gove We have taken extreme caution in protecting the system while using libraries posted from outside

          1) user has to enable it explicitly from command line
          2) user can optionally choose to enforce that the binaries are signed by a certain private key

          Show
          noble.paul Noble Paul added a comment - - edited I believe all components should be dynamically loadable. Kevin Risden , please explain (in description) how users can register a streaming expression plugin and use it if the class is loaded from a runtime jar Dennis Gove We have taken extreme caution in protecting the system while using libraries posted from outside 1) user has to enable it explicitly from command line 2) user can optionally choose to enforce that the binaries are signed by a certain private key
          Hide
          risdenk Kevin Risden added a comment -

          Noble Paul Added in the description how to register a custom streaming expression with a runtime jar.

          Show
          risdenk Kevin Risden added a comment - Noble Paul Added in the description how to register a custom streaming expression with a runtime jar.
          Hide
          noble.paul Noble Paul added a comment -

          Thanks Kevin Risden

          I guess we should cut down he steps more . I plan to implement SOLR-7955 so that one step to create the ..system collection is gone. Can we have a well known package such as solr.streamingexp.customexpr automatically map to an expressible called customexpr. So, user can effectively eliminate that step as well.

          Show
          noble.paul Noble Paul added a comment - Thanks Kevin Risden I guess we should cut down he steps more . I plan to implement SOLR-7955 so that one step to create the ..system collection is gone. Can we have a well known package such as solr.streamingexp.customexpr automatically map to an expressible called customexpr . So, user can effectively eliminate that step as well.
          Hide
          risdenk Kevin Risden added a comment -

          I think the current steps aren't that bad. We can debate changing the process in a different JIRA. This was opened as a one line change that enables jars to be loaded. Without the change jars can't be loaded at all from the blob store.

          Show
          risdenk Kevin Risden added a comment - I think the current steps aren't that bad. We can debate changing the process in a different JIRA. This was opened as a one line change that enables jars to be loaded. Without the change jars can't be loaded at all from the blob store.
          Hide
          noble.paul Noble Paul added a comment -

          Totally agree. Th is ticket should limit the scope to the description and we should close this

          Show
          noble.paul Noble Paul added a comment - Totally agree. Th is ticket should limit the scope to the description and we should close this
          Hide
          dpgove Dennis Gove added a comment -

          I think this is a good addition.

          Show
          dpgove Dennis Gove added a comment - I think this is a good addition.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit db987b810cd9acaae3fae891ef07258b09d4d94b in lucene-solr's branch refs/heads/master from Kevin Risden
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=db987b8 ]

          SOLR-10087: StreamHandler should be able to use runtimeLib jars

          Show
          jira-bot ASF subversion and git services added a comment - Commit db987b810cd9acaae3fae891ef07258b09d4d94b in lucene-solr's branch refs/heads/master from Kevin Risden [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=db987b8 ] SOLR-10087 : StreamHandler should be able to use runtimeLib jars
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e457613e6ec9fa258f9fd170b00dec58547e51e6 in lucene-solr's branch refs/heads/branch_6x from Kevin Risden
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e457613 ]

          SOLR-10087: StreamHandler should be able to use runtimeLib jars

          Show
          jira-bot ASF subversion and git services added a comment - Commit e457613e6ec9fa258f9fd170b00dec58547e51e6 in lucene-solr's branch refs/heads/branch_6x from Kevin Risden [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e457613 ] SOLR-10087 : StreamHandler should be able to use runtimeLib jars
          Hide
          noble.paul Noble Paul added a comment -

          Kevin Risden The JDBCStream class load drivers from outside. It would be nice if you could load it from runtimelib jars as well

          Show
          noble.paul Noble Paul added a comment - Kevin Risden The JDBCStream class load drivers from outside. It would be nice if you could load it from runtimelib jars as well
          Hide
          risdenk Kevin Risden added a comment -

          Noble Paul - Can you open a new JIRA issue for that?

          Show
          risdenk Kevin Risden added a comment - Noble Paul - Can you open a new JIRA issue for that?
          Hide
          noble.paul Noble Paul added a comment -
          Show
          noble.paul Noble Paul added a comment - Kevin Risden opened SOLR-10105
          Hide
          ctargett Cassandra Targett added a comment -

          Kevin Risden: are the instructions in the description of this issue still accurate after it was committed? Asking for the Ref Guide, unless perhaps you think we should skip documenting this.

          Show
          ctargett Cassandra Targett added a comment - Kevin Risden : are the instructions in the description of this issue still accurate after it was committed? Asking for the Ref Guide, unless perhaps you think we should skip documenting this.
          Hide
          risdenk Kevin Risden added a comment -

          Cassandra Targett - I'm 90% sure the instructions are still accurate. I haven't had a chance to try it with the new Solr release though.

          Show
          risdenk Kevin Risden added a comment - Cassandra Targett - I'm 90% sure the instructions are still accurate. I haven't had a chance to try it with the new Solr release though.
          Hide
          ctargett Cassandra Targett added a comment -

          Kevin Risden - I just tried it with your example project, and everything goes smoothly until creating the custom expression with create-expressible. I get an error like:

          msg=Error trying to proxy request for url: http://192.168.0.110:7574/solr/test/config,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://192.168.0.110:7574/solr/test/config
          

          And then in the stack trace: Caused by: java.net.SocketException: Bad file descriptor. I'm using 6.5.0 that was just released. I have a feeling this is something I'm doing wrong since I'm not well-versed in blob store, but adding info in case you see a bug.

          Show
          ctargett Cassandra Targett added a comment - Kevin Risden - I just tried it with your example project, and everything goes smoothly until creating the custom expression with create-expressible . I get an error like: msg=Error trying to proxy request for url: http: //192.168.0.110:7574/solr/test/config,trace=org.apache.solr.common.SolrException: Error trying to proxy request for url: http://192.168.0.110:7574/solr/test/config And then in the stack trace: Caused by: java.net.SocketException: Bad file descriptor . I'm using 6.5.0 that was just released. I have a feeling this is something I'm doing wrong since I'm not well-versed in blob store, but adding info in case you see a bug.
          Hide
          risdenk Kevin Risden added a comment -

          Hmmm I'll try to try it later today if I get a few minutes. I'll update by github project to use the new 6.5.0 release and remove the custom jar I had placed in the project.

          Show
          risdenk Kevin Risden added a comment - Hmmm I'll try to try it later today if I get a few minutes. I'll update by github project to use the new 6.5.0 release and remove the custom jar I had placed in the project.
          Hide
          risdenk Kevin Risden added a comment -

          Cassandra Targett - I updated the repo https://github.com/risdenk/solr_custom_streaming_expressions to use Solr 6.5.0 and had no issues. Bad file descriptor is a weird error to be getting.

          Show
          risdenk Kevin Risden added a comment - Cassandra Targett - I updated the repo https://github.com/risdenk/solr_custom_streaming_expressions to use Solr 6.5.0 and had no issues. Bad file descriptor is a weird error to be getting.

            People

            • Assignee:
              risdenk Kevin Risden
              Reporter:
              risdenk Kevin Risden
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development