Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Make it possible to define paramsets and use them directly in requests

      example

      curl http://localhost:8983/solr/collection1/config/params -H 'Content-type:application/json'  -d '{
      "set" : {"x": {
                    "a":"A val",
                    "b": "B val"}
                 },
      "set" : {"y": {
                     "x":"X val",
                     "Y": "Y val"}
                 },
      "update" : {"y": {
                     "x":"X val modified"}
                 },
      
      "delete" : "z"
      }'
      #do a GET to view all the configured params
      curl http://localhost:8983/solr/collection1/config/params
      
      #or  GET with a specific name to get only one set of params
      curl http://localhost:8983/solr/collection1/config/params/x
      

      This data will be stored in conf/params.json

      This is used requesttime and adding/editing params will not result in core reload and it will have no impact on the performance

      example usage http://localhost/solr/collection/select?useParams=x,y

      or it can be directly configured with a request handler as follows

      <requestHandler name="/dump1" class="DumpRequestHandler" useParams="x"/>
      

      useParams specified in request overrides the one specified in requestHandler

      A more realistic example

      curl http://localhost:8983/solr/collection1/config/params -H 'Content-type:application/json'  -d '{
      "set":{"query":{
          "defType":"edismax",
          "q.alt":"*:*",
          "rows":10,
          "fl":"*,score"  },
        "facets":{
          "facet":"on",
          "facet.mincount": 1
        },
       "velocity":{
         "wt": "velocity",
         "v.template":"browse",
         "v.layout": "layout"
       }
      }
      }
      

      and use all of them directly is a requesthandler

        <requestHandler name="/browse" class="solr.SearchHandler" useParams="query,facets,velocity,browse"/>  
      
      1. SOLR-6770.patch
        40 kB
        Noble Paul
      2. SOLR-6770.patch
        37 kB
        Noble Paul
      3. SOLR-6770.patch
        33 kB
        Noble Paul

        Activity

        Hide
        David Smiley added a comment -

        +1 Nice! Looking forward to this.

        Show
        David Smiley added a comment - +1 Nice! Looking forward to this.
        Hide
        Alexandre Rafalovitch added a comment -

        Would these be default params only? Or compulsory/added params as well?

        Show
        Alexandre Rafalovitch added a comment - Would these be default params only? Or compulsory/added params as well?
        Hide
        Noble Paul added a comment -

        These are shorthands for request params . it is equivalent to defaults but the component will not have access to these during init() . So it is purely a request time construct

        Show
        Noble Paul added a comment - These are shorthands for request params . it is equivalent to defaults but the component will not have access to these during init() . So it is purely a request time construct
        Hide
        Alexandre Rafalovitch added a comment -

        What about in your second example where you are defining them on the requestHandler itself? Does it become just a second set of default to layer on top of the set defined in-place. What order does it apply in if both are present?

        Show
        Alexandre Rafalovitch added a comment - What about in your second example where you are defining them on the requestHandler itself? Does it become just a second set of default to layer on top of the set defined in-place. What order does it apply in if both are present?
        Hide
        Noble Paul added a comment - - edited

        The second option is an shorthand if you don't want to pass it with every request and again those parameters will not be available during init()

        The construct allows you to apply more than one param sets. I can say useParam=x,y,z and in that case all are applied in the order they are specified (the way defaults are applied currently) . If the request has a useParam value , the one specified in the requestHandler is ignored.

        Show
        Noble Paul added a comment - - edited The second option is an shorthand if you don't want to pass it with every request and again those parameters will not be available during init() The construct allows you to apply more than one param sets. I can say useParam=x,y,z and in that case all are applied in the order they are specified (the way defaults are applied currently) . If the request has a useParam value , the one specified in the requestHandler is ignored.
        Hide
        Shalin Shekhar Mangar added a comment -

        This looks nice!

        1. Can we please use "params" or "value" instead of "val" in the json commands?
        2. Specifying the content-type would be optional, right?
        Show
        Shalin Shekhar Mangar added a comment - This looks nice! Can we please use "params" or "value" instead of "val" in the json commands? Specifying the content-type would be optional, right?
        Hide
        Noble Paul added a comment - - edited

        yes we can change the key

        If we don't specify the content type curl sets it to application/x-www-form-urlencoded so we can't omit it in the examples

        Show
        Noble Paul added a comment - - edited yes we can change the key If we don't specify the content type curl sets it to application/x-www-form-urlencoded so we can't omit it in the examples
        Hide
        Noble Paul added a comment -

        feature done, tests not done

        Show
        Noble Paul added a comment - feature done, tests not done
        Hide
        Noble Paul added a comment -

        I'm planning to commit this soon

        Show
        Noble Paul added a comment - I'm planning to commit this soon
        Hide
        Yonik Seeley added a comment - - edited

        The singular "param" in "useParam" sounds a little weird since you're actually specifying a set of params. How about "useParams"?

        #do a GET to view all the configured params
        curl http://localhost:8983/solr/collection1/config/params
        
        #or  GET with a specific name to get only one set of params
        curl http://localhost:8983/solr/collection1/config/params/x
        

        These feels very rest-like... is there a reason we're not going more rest-like for the updates as well?

        curl http://localhost:8983/solr/collection1/config/params/x -d '
        {
                      "a":"A val",
                      "b": "B val"
        }
        '
        
        OR...
        
        curl -XPUT http://localhost:8983/solr/collection1/config/params -d '
        {
          "x" : {
               "a":"A val",
               "b": "B val"
          }
        }
        '
        

        I suppose it's because of the desire to be able to specify "create" vs "update" (and I assume the latter changes a param set rather than overwrites it?).
        Is there another "resty" way to specify that distinction?

        Even if we keep updates of the form:

        "create" : {"name" ,"x",
                    "params": {
                      "a":"A val",
                      "b": "B val"}
        

        We could actually eliminate the other names of "name" and "params" with a structure like:

        "create" : {
          "x": {
            "a":"A val",
            "b": "B val"
          }
        }
        

        Also, if "create" can overwrite the previous param set, then "set" might be a better verb.

        Show
        Yonik Seeley added a comment - - edited The singular "param" in "useParam" sounds a little weird since you're actually specifying a set of params. How about "useParams"? # do a GET to view all the configured params curl http: //localhost:8983/solr/collection1/config/params #or GET with a specific name to get only one set of params curl http: //localhost:8983/solr/collection1/config/params/x These feels very rest-like... is there a reason we're not going more rest-like for the updates as well? curl http: //localhost:8983/solr/collection1/config/params/x -d ' { "a" : "A val" , "b" : "B val" } ' OR... curl -XPUT http: //localhost:8983/solr/collection1/config/params -d ' { "x" : { "a" : "A val" , "b" : "B val" } } ' I suppose it's because of the desire to be able to specify "create" vs "update" (and I assume the latter changes a param set rather than overwrites it?). Is there another "resty" way to specify that distinction? Even if we keep updates of the form: "create" : { "name" , "x" , "params" : { "a" : "A val" , "b" : "B val" } We could actually eliminate the other names of "name" and "params" with a structure like: "create" : { "x" : { "a" : "A val" , "b" : "B val" } } Also, if "create" can overwrite the previous param set, then "set" might be a better verb.
        Hide
        Noble Paul added a comment - - edited

        The singular "param" in "useParam"

        I'm still ambivalent about it. I shall make it useParams

        These feels very rest-like... is there a reason we're not going more rest-like for the updates as well?

        This could have followed full REST. But, then, we have a lot of APIs schema/config which support only GET/POST. I wanted this to be fully consistent with those APIs.

        the verbs available are create ,update,modify and delete

        I'm still open to renaming the verbs .

        Show
        Noble Paul added a comment - - edited The singular "param" in "useParam" I'm still ambivalent about it. I shall make it useParams These feels very rest-like... is there a reason we're not going more rest-like for the updates as well? This could have followed full REST. But, then, we have a lot of APIs schema/config which support only GET/POST. I wanted this to be fully consistent with those APIs. the verbs available are create , update , modify and delete I'm still open to renaming the verbs .
        Hide
        Noble Paul added a comment - - edited

        We could actually eliminate the other names of "name" and "params" with a structure like:

        I actually started with that . I am worried how to support any metadata if I wish to add them in future ?

        "create" : {
          "x": {
            "": {"meta-key1" : "val"},
            "a":"A val",
            "b": "B val"
          }
        }
        
        Show
        Noble Paul added a comment - - edited We could actually eliminate the other names of "name" and "params" with a structure like: I actually started with that . I am worried how to support any metadata if I wish to add them in future ? "create" : { "x" : { "": {" meta-key1 " : " val"}, "a" : "A val" , "b" : "B val" } }
        Hide
        Noble Paul added a comment -

        OK , I'll change the syntax and post a new patch

        Show
        Noble Paul added a comment - OK , I'll change the syntax and post a new patch
        Hide
        Noble Paul added a comment -

        changed the syntax . The description is modified accordingly

        Show
        Noble Paul added a comment - changed the syntax . The description is modified accordingly
        Hide
        ASF subversion and git services added a comment -

        Commit 1647748 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1647748 ]

        SOLR-6770 Add/edit param sets and use them in requests

        Show
        ASF subversion and git services added a comment - Commit 1647748 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1647748 ] SOLR-6770 Add/edit param sets and use them in requests
        Hide
        Yonik Seeley added a comment -

        Do we really need separate "create" and "update" actions? (and I was initially confused about the difference between "update" and "modify").
        "create" doesn't seem so useful if "update" can do the same thing without erroring if it already exists.

        Show
        Yonik Seeley added a comment - Do we really need separate "create" and "update" actions? (and I was initially confused about the difference between "update" and "modify"). "create" doesn't seem so useful if "update" can do the same thing without erroring if it already exists.
        Hide
        Noble Paul added a comment -

        I have separate commands to avoid accidental overwrite. It is just to idiot proof the system.

        Show
        Noble Paul added a comment - I have separate commands to avoid accidental overwrite. It is just to idiot proof the system.
        Hide
        Yonik Seeley added a comment -

        I have separate commands to avoid accidental overwrite. It is just to idiot proof the system.

        I understand the rational, but I just wonder if it makes it easier or harder to use in practice. This actually creates another mode of failure.
        As an example, a common "idiot"/newbie mistake to make may now be:

        1) send in a "create" command
        2) try it out... yay, it works.
        3) modify the command a bit and send it in again... failure? Arg... I have to use a different command if it already exists?

        Look at our APIs for adding documents... by default it's a create that replaces any existing document.

        Obviously, the right default semantics depend on the API and expected use-cases. But I can't think of realistic use-cases when I want to add a param set but I want it to fail if it already exists.

        Show
        Yonik Seeley added a comment - I have separate commands to avoid accidental overwrite. It is just to idiot proof the system. I understand the rational, but I just wonder if it makes it easier or harder to use in practice. This actually creates another mode of failure. As an example, a common "idiot"/newbie mistake to make may now be: 1) send in a "create" command 2) try it out... yay, it works. 3) modify the command a bit and send it in again... failure? Arg... I have to use a different command if it already exists? Look at our APIs for adding documents... by default it's a create that replaces any existing document. Obviously, the right default semantics depend on the API and expected use-cases. But I can't think of realistic use-cases when I want to add a param set but I want it to fail if it already exists.
        Hide
        Noble Paul added a comment - - edited

        But I can't think of realistic use-cases when I want to add a param set but I want it to fail if it already exists.

        Actually , the usecase is, I'm trying to update a paramset and I made a typo in the name . Now, I have two paramsets and the user thought he was right and he gets weird results

        To be honest I can go either way

        Show
        Noble Paul added a comment - - edited But I can't think of realistic use-cases when I want to add a param set but I want it to fail if it already exists. Actually , the usecase is, I'm trying to update a paramset and I made a typo in the name . Now, I have two paramsets and the user thought he was right and he gets weird results To be honest I can go either way
        Hide
        ASF subversion and git services added a comment -
        Show
        ASF subversion and git services added a comment - Commit 1647829 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1647829 ] SOLR-6770 fixing test failures http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/2013/
        Hide
        Yonik Seeley added a comment -

        Actually , the usecase is, I'm trying to update a paramset and I made a typo in the name . Now, I have two paramsets and the user thought he was right and he gets weird results

        Urgg... does this mean that I can't just use "update" if I don't care if I'm overwriting a previous one or not? If that's true, I've mis-interpreted these parameters yet again.

        This doesn't really protect against anything. Someone could just as easily make a typo in one of the parameters or one of the values. The name isn't special. This just makes it harder for people not making typing errors.

        For example, if I make a script to set a bunch of params periodically, I just want one way to do it. I don't want to have to do it one way the first time and then a different way every subsequent time.

        Show
        Yonik Seeley added a comment - Actually , the usecase is, I'm trying to update a paramset and I made a typo in the name . Now, I have two paramsets and the user thought he was right and he gets weird results Urgg... does this mean that I can't just use "update" if I don't care if I'm overwriting a previous one or not? If that's true, I've mis-interpreted these parameters yet again. This doesn't really protect against anything. Someone could just as easily make a typo in one of the parameters or one of the values. The name isn't special. This just makes it harder for people not making typing errors. For example, if I make a script to set a bunch of params periodically, I just want one way to do it. I don't want to have to do it one way the first time and then a different way every subsequent time.
        Hide
        Alexandre Rafalovitch added a comment -

        Is there a result status (created, updated, deleted, failed, etc)? Might be a way to unify the API yet show the difference in treatment.

        Show
        Alexandre Rafalovitch added a comment - Is there a result status (created, updated, deleted, failed, etc)? Might be a way to unify the API yet show the difference in treatment.
        Hide
        David Smiley added a comment -

        +1 to this point. Like with documents, I want to add if not there or update if it is there. Perhaps the response might let me know what happened if I'm curious.

        Show
        David Smiley added a comment - +1 to this point. Like with documents, I want to add if not there or update if it is there. Perhaps the response might let me know what happened if I'm curious.
        Hide
        Noble Paul added a comment -

        OK, So let's make the commands

        • set : create if does not exist. replace if exists
        • update : merge the map with new values
        • delete
        Show
        Noble Paul added a comment - OK, So let's make the commands set : create if does not exist. replace if exists update : merge the map with new values delete
        Hide
        David Smiley added a comment -

        +1

        Show
        David Smiley added a comment - +1
        Hide
        ASF subversion and git services added a comment -

        Commit 1647962 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1647962 ]

        SOLR-6770 changed command names to set, update

        Show
        ASF subversion and git services added a comment - Commit 1647962 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1647962 ] SOLR-6770 changed command names to set, update
        Hide
        ASF subversion and git services added a comment -

        Commit 1648689 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1648689 ]

        SOLR-6801 more tests , SOLR-6770 refactored code around

        Show
        ASF subversion and git services added a comment - Commit 1648689 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1648689 ] SOLR-6801 more tests , SOLR-6770 refactored code around
        Hide
        ASF subversion and git services added a comment -

        Commit 1648847 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1648847 ]

        SOLR-6770: Add/edit param sets and use them in Requests

        Show
        ASF subversion and git services added a comment - Commit 1648847 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1648847 ] SOLR-6770 : Add/edit param sets and use them in Requests
        Hide
        ASF subversion and git services added a comment -

        Commit 1649542 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1649542 ]

        SOLR-6770 wrong file name in standalone mode

        Show
        ASF subversion and git services added a comment - Commit 1649542 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1649542 ] SOLR-6770 wrong file name in standalone mode
        Hide
        ASF subversion and git services added a comment -

        Commit 1649543 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1649543 ]

        SOLR-6770 wrong file name in standalone mode

        Show
        ASF subversion and git services added a comment - Commit 1649543 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1649543 ] SOLR-6770 wrong file name in standalone mode
        Hide
        ASF subversion and git services added a comment -

        Commit 1649551 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1649551 ]

        SOLR-6770 Added testcase for standalone mode and fixed a reload bug

        Show
        ASF subversion and git services added a comment - Commit 1649551 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1649551 ] SOLR-6770 Added testcase for standalone mode and fixed a reload bug
        Hide
        ASF subversion and git services added a comment -

        Commit 1649553 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1649553 ]

        SOLR-6770 Added testcase for standalone mode and fixed a reload bug

        Show
        ASF subversion and git services added a comment - Commit 1649553 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1649553 ] SOLR-6770 Added testcase for standalone mode and fixed a reload bug
        Hide
        ASF subversion and git services added a comment -

        Commit 1651308 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1651308 ]

        SOLR-6770 Optimize tests to avoid duplicate RE[BST calls

        Show
        ASF subversion and git services added a comment - Commit 1651308 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1651308 ] SOLR-6770 Optimize tests to avoid duplicate RE[BST calls
        Hide
        ASF subversion and git services added a comment -

        Commit 1651316 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1651316 ]

        SOLR-6770 Optimize tests to avoid duplicate REST calls

        Show
        ASF subversion and git services added a comment - Commit 1651316 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1651316 ] SOLR-6770 Optimize tests to avoid duplicate REST calls
        Hide
        ASF subversion and git services added a comment -

        Commit 1652134 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1652134 ]

        SOLR-6770 mask the useParams aftger expanding it

        Show
        ASF subversion and git services added a comment - Commit 1652134 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1652134 ] SOLR-6770 mask the useParams aftger expanding it
        Hide
        ASF subversion and git services added a comment -

        Commit 1652137 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1652137 ]

        SOLR-6770 mask the useParams after expanding it

        Show
        ASF subversion and git services added a comment - Commit 1652137 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1652137 ] SOLR-6770 mask the useParams after expanding it
        Hide
        ASF subversion and git services added a comment -

        Commit 1652153 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1652153 ]

        SOLR-6770 accidentally commented out a test

        Show
        ASF subversion and git services added a comment - Commit 1652153 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1652153 ] SOLR-6770 accidentally commented out a test
        Hide
        ASF subversion and git services added a comment -

        Commit 1652155 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1652155 ]

        SOLR-6770 accidentally commented out a test

        Show
        ASF subversion and git services added a comment - Commit 1652155 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1652155 ] SOLR-6770 accidentally commented out a test
        Hide
        ASF subversion and git services added a comment -

        Commit 1652159 from Noble Paul in branch 'dev/branches/lucene_solr_5_0'
        [ https://svn.apache.org/r1652159 ]

        SOLR-6770 mask the useParams after expanding it

        Show
        ASF subversion and git services added a comment - Commit 1652159 from Noble Paul in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1652159 ] SOLR-6770 mask the useParams after expanding it
        Hide
        Alexandre Rafalovitch added a comment -

        Do we have a real example of this, rather than a/b/c? I am not entirely sure how to use/test these in the end.

        Show
        Alexandre Rafalovitch added a comment - Do we have a real example of this, rather than a/b/c? I am not entirely sure how to use/test these in the end.
        Hide
        Noble Paul added a comment -

        The current trunk implements the /browse feature using useParams

        I'll post another real world example

        Show
        Noble Paul added a comment - The current trunk implements the /browse feature using useParams I'll post another real world example
        Hide
        Anshum Gupta added a comment -

        Bulk close after 5.0 release.

        Show
        Anshum Gupta added a comment - Bulk close after 5.0 release.
        Hide
        ASF subversion and git services added a comment -

        Commit 1666816 from Yonik Seeley in branch 'dev/trunk'
        [ https://svn.apache.org/r1666816 ]

        SOLR-6770: reformat - fix bad indentation and funky formatting

        Show
        ASF subversion and git services added a comment - Commit 1666816 from Yonik Seeley in branch 'dev/trunk' [ https://svn.apache.org/r1666816 ] SOLR-6770 : reformat - fix bad indentation and funky formatting
        Hide
        ASF subversion and git services added a comment -

        Commit 1666817 from Yonik Seeley in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1666817 ]

        SOLR-6770: reformat - fix bad indentation and funky formatting

        Show
        ASF subversion and git services added a comment - Commit 1666817 from Yonik Seeley in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1666817 ] SOLR-6770 : reformat - fix bad indentation and funky formatting

          People

          • Assignee:
            Noble Paul
            Reporter:
            Noble Paul
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development