Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      The concept of solrconfig editing is split into multiple pieces . This issue is about registering components and uploading binaries through an API.

      This supports multiple operations

      • commands 'create-requesthandler', "update-requesthandler","delete-requesthandler" which can set the configuration of a component . This configuration will be saved inside the configoverlay.json

      The components has to be available in the classpath of all nodes.
      example for registering a component

      curl http://localhost:8983/solr/collection1/config -H 'Content-type:application/json'  -d '{
      "create-requesthandler" : {"name": "/mypath" ,
                                                "class":"com.mycomponent.ClassName" , 
                                                 "defaults":{"x":"y" ," a":"b"},
                                                 "useParams":"x"
                                               },
      "update-requesthandler" :{"name": "/mypath" ,
                                                 "class":"com.mycomponent.ClassName" ,
                                                 "useParams":"y" ,
                                                 "defaults":{"x":"y" ," a":"b"}
                                               },
      "delete-requesthandler" :"/mypath" 
      }'
      

        Issue Links

          Activity

          Hide
          Noble Paul added a comment - - edited

          Done with testcases

          Show
          Noble Paul added a comment - - edited Done with testcases
          Hide
          Noble Paul added a comment -

          The scope of this ticket is reduced to just registering existing components. Loading external components will be tackled in SOLR-6801

          Show
          Noble Paul added a comment - The scope of this ticket is reduced to just registering existing components. Loading external components will be tackled in SOLR-6801
          Hide
          ASF subversion and git services added a comment -

          Commit 1642141 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1642141 ]

          SOLR-6607 Managing requesthandlers through API

          Show
          ASF subversion and git services added a comment - Commit 1642141 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1642141 ] SOLR-6607 Managing requesthandlers through API
          Hide
          ASF subversion and git services added a comment -

          Commit 1642143 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1642143 ]

          SOLR-6607 Managing requesthandlers through API

          Show
          ASF subversion and git services added a comment - Commit 1642143 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1642143 ] SOLR-6607 Managing requesthandlers through API
          Hide
          ASF subversion and git services added a comment -

          Commit 1642151 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1642151 ]

          SOLR-6607 /config path should show overridden requesthandler

          Show
          ASF subversion and git services added a comment - Commit 1642151 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1642151 ] SOLR-6607 /config path should show overridden requesthandler
          Hide
          ASF subversion and git services added a comment -

          Commit 1642153 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1642153 ]

          SOLR-6607 /config path should show overridden requesthandler

          Show
          ASF subversion and git services added a comment - Commit 1642153 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1642153 ] SOLR-6607 /config path should show overridden requesthandler
          Hide
          Alexandre Rafalovitch added a comment -

          Would it be possible - if not done already - for when these combined request handlers returned, there is a synthetic attribute which says whether they were loaded from source, from overlay (only), or from overlay as an over-ride.

          Having such flag will significantly simplify troubleshooting for both user and Solr developer adding more features later.

          Show
          Alexandre Rafalovitch added a comment - Would it be possible - if not done already - for when these combined request handlers returned, there is a synthetic attribute which says whether they were loaded from source, from overlay (only), or from overlay as an over-ride. Having such flag will significantly simplify troubleshooting for both user and Solr developer adding more features later.
          Hide
          Noble Paul added a comment -

          Currently , you can hit /config/overlay to know the request handlers specified in overlay. And /config/requestHandler can give you all request handlers registered in the core.

          Currently , there are three places a requesthandler can be loaded from

          • registered in overlay
          • registered in solrconfig.xml
          • implicit ones , registered in code

          And the priority is in that order .

          However, we are planning to add a new a new endpoint /<requesthandlername>/_meta which gives the details of the requesthandler .

          Do you think it is necessary to know from where it is loaded in the /config end point itself ?

          Show
          Noble Paul added a comment - Currently , you can hit /config/overlay to know the request handlers specified in overlay. And /config/requestHandler can give you all request handlers registered in the core. Currently , there are three places a requesthandler can be loaded from registered in overlay registered in solrconfig.xml implicit ones , registered in code And the priority is in that order . However, we are planning to add a new a new endpoint /<requesthandlername>/_meta which gives the details of the requesthandler . Do you think it is necessary to know from where it is loaded in the /config end point itself ?
          Hide
          Alexandre Rafalovitch added a comment -

          I think it is very important to have unified way to get all the different permutations of ground truth. Prior to these - exciting - new developments, there was one place to find the ground truth - in solrconfig.xml. You could grep the file and get everything you needed (well, except for admin ones, but they were mentioned in the comments).

          But now, we have a lot more handlers that are implicit plus the hardcoded ones and the overlay ones. If there is no way to get all of the information (including the source of the definition) in one place, there will be confusion down the line. From users and - I suspect - tool writers.

          So, I think a single end-point with parameters to differentiate the source would be nice. The parameters could be ?source=all, source=implicit, source=defined, source=overlay as well as status=active, status=overriden and status=disabled/inactive (maybe). By default, it would be source=all and status=active. These two parameters would be included for each entry.

          So, if I have /selectAR in solrconfig.xml and it's also overriden in the overlay then it will be returned as source=overlay, status=active when I request with source=all, but as source=defined, status=overriden when I request with source=defined.

          Does it make sense?

          Show
          Alexandre Rafalovitch added a comment - I think it is very important to have unified way to get all the different permutations of ground truth . Prior to these - exciting - new developments, there was one place to find the ground truth - in solrconfig.xml. You could grep the file and get everything you needed (well, except for admin ones, but they were mentioned in the comments). But now, we have a lot more handlers that are implicit plus the hardcoded ones and the overlay ones. If there is no way to get all of the information (including the source of the definition) in one place, there will be confusion down the line. From users and - I suspect - tool writers. So, I think a single end-point with parameters to differentiate the source would be nice. The parameters could be ?source=all, source=implicit, source=defined, source=overlay as well as status=active, status=overriden and status=disabled/inactive (maybe). By default, it would be source=all and status=active. These two parameters would be included for each entry. So, if I have /selectAR in solrconfig.xml and it's also overriden in the overlay then it will be returned as source=overlay , status=active when I request with source=all , but as source=defined , status=overriden when I request with source=defined . Does it make sense?
          Hide
          Noble Paul added a comment -

          how about adding extra information to the /config end point

          example

          "/analysis/field": {
          "_meta_" : {
                            "source" : "overlay", // alternatively, implicit, solrconfig.xml
                            "overrides" : " solrconfig.xml"
                            }
          "startup": "lazy",
          "name": "/analysis/field",
          "class": "solr.FieldAnalysisRequestHandler"
          }
          
          Show
          Noble Paul added a comment - how about adding extra information to the /config end point example "/analysis/field" : { "_meta_" : { "source" : "overlay" , // alternatively, implicit, solrconfig.xml "overrides" : " solrconfig.xml" } "startup" : "lazy" , "name" : "/analysis/field" , "class" : "solr.FieldAnalysisRequestHandler" }
          Hide
          Noble Paul added a comment -

          So, I think a single end-point with parameters to differentiate the source would be nice

          The objective pf /config endpoint is to give the definitive schema that is in use now

          For checking the overlay the endpoint is /config/overlay. Probably we can add more end points for solrconfig.xml

          Show
          Noble Paul added a comment - So, I think a single end-point with parameters to differentiate the source would be nice The objective pf /config endpoint is to give the definitive schema that is in use now For checking the overlay the endpoint is /config/overlay . Probably we can add more end points for solrconfig.xml
          Hide
          Alexandre Rafalovitch added a comment -

          I don't have a terribly strong opinion on the specific setup. As long as it can answer two questions easily:
          1. What handlers do I actually have
          2. Why do I have those specific handlers (where are they defined and whether they override/are overriden).

          The above proposal seems to cover those two questions.

          Show
          Alexandre Rafalovitch added a comment - I don't have a terribly strong opinion on the specific setup. As long as it can answer two questions easily: 1. What handlers do I actually have 2. Why do I have those specific handlers (where are they defined and whether they override/are overriden). The above proposal seems to cover those two questions.
          Hide
          Alexandre Rafalovitch added a comment -

          Ugh, just realized. Does this also cover handleSelect=true and reports it (the fourth location? actually even messier with qt=).

          Also related: SOLR-6807 and SOLR-3346

          Show
          Alexandre Rafalovitch added a comment - Ugh, just realized. Does this also cover handleSelect=true and reports it (the fourth location? actually even messier with qt=). Also related: SOLR-6807 and SOLR-3346
          Hide
          Noble Paul added a comment -

          I don't think qt= matter here. handleSelect is not a problem either. It is using the same handler that is defined somewhere using one of the existing mechanisms

          Show
          Noble Paul added a comment - I don't think qt= matter here. handleSelect is not a problem either. It is using the same handler that is defined somewhere using one of the existing mechanisms
          Hide
          Yonik Seeley added a comment -

          You gave a good example of what the API looks like, but not so much what it does.
          Does it work in solr-cloud mode? In stand-alone mode?
          Are the changes persisted somehow in both of these modes? If so, what's the mechanism. If not, is there a plan for that?

          Show
          Yonik Seeley added a comment - You gave a good example of what the API looks like, but not so much what it does. Does it work in solr-cloud mode? In stand-alone mode? Are the changes persisted somehow in both of these modes? If so, what's the mechanism. If not, is there a plan for that?
          Hide
          Noble Paul added a comment -

          Does it work in solr-cloud mode? In stand-alone mode?

          Both.

          Are the changes persisted somehow in both of these modes? If so, what's the mechanism. If not, is there a plan for that?

          In both cases changes are persisted.

          for standalone node , it is written to the conf dir

          for solrcloud it is written to zk

          Show
          Noble Paul added a comment - Does it work in solr-cloud mode? In stand-alone mode? Both. Are the changes persisted somehow in both of these modes? If so, what's the mechanism. If not, is there a plan for that? In both cases changes are persisted. for standalone node , it is written to the conf dir for solrcloud it is written to zk
          Hide
          ASF subversion and git services added a comment -

          Commit 1649821 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1649821 ]

          SOLR-6801 addressing test failures, SOLR-6607 supporting all initArgs for requesthandlers

          Show
          ASF subversion and git services added a comment - Commit 1649821 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1649821 ] SOLR-6801 addressing test failures, SOLR-6607 supporting all initArgs for requesthandlers
          Hide
          ASF subversion and git services added a comment -

          Commit 1649826 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1649826 ]

          SOLR-6801 addressing test failures, SOLR-6607 supporting all initArgs for requesthandlers

          Show
          ASF subversion and git services added a comment - Commit 1649826 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1649826 ] SOLR-6801 addressing test failures, SOLR-6607 supporting all initArgs for requesthandlers
          Hide
          Alexandre Rafalovitch added a comment -

          What happened with the meta implementation for the ground truth?

          Now that /update is in the explicit configuration, I cannot seem to find that indicated when I do *curl http://localhost:8983/solr/techproducts/config/requestHandler*

          I talked about the ground truth in this issue on the November 28th. I can't find what addresses it. Is there a different JIRA that I am missing perhaps? Without the "ground truth" it is very hard to debug things.

          Show
          Alexandre Rafalovitch added a comment - What happened with the meta implementation for the ground truth? Now that /update is in the explicit configuration, I cannot seem to find that indicated when I do *curl http://localhost:8983/solr/techproducts/config/requestHandler* I talked about the ground truth in this issue on the November 28th. I can't find what addresses it. Is there a different JIRA that I am missing perhaps? Without the "ground truth" it is very hard to debug things.
          Hide
          Noble Paul added a comment - - edited

          I got sucked into the bug fix mode. Let's track it in a separate issue SOLR-6964

          Show
          Noble Paul added a comment - - edited I got sucked into the bug fix mode. Let's track it in a separate issue SOLR-6964
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.

            People

            • Assignee:
              Noble Paul
              Reporter:
              Noble Paul
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development