Solr
  1. Solr
  2. SOLR-6365

specify appends, defaults, invariants outside of the component

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      The components are configured in solrconfig.xml mostly for specifying these extra parameters. If we separate these out, we can avoid specifying the components altogether and make solrconfig much simpler. Eventually we want users to see all functions as paths instead of components and control these params from outside , through an API and persisted in ZK

      objectives :

      • define standard components implicitly and let users override some params only
      • reuse standard params across components
      • define multiple param sets and mix and match these params at request time

      example

      <!-- use json for all paths and _txt as the default search field-->
      <initParams name="global" path="/**">
        <lst name="defaults">
           <str name="wt">json</str>
           <str name="df">_txt</str>
        </lst>
      </initParams>
      

      other examples

      <initParams name="a" path="/dump3,/root/*,/root1/**">
          <lst name="defaults">
            <str name="a">A</str>
          </lst>
          <lst name="invariants">
            <str name="b">B</str>
          </lst>
          <lst name="appends">
            <str name="c">C</str>
          </lst>
        </initParams>
        <requestHandler name="/dump3" class="DumpRequestHandler"/>
        <requestHandler name="/dump4" class="DumpRequestHandler"/>
        <requestHandler name="/root/dump5" class="DumpRequestHandler"/>
        <requestHandler name="/root1/anotherlevel/dump6" class="DumpRequestHandler"/>
        <requestHandler name="/dump1" class="DumpRequestHandler" initParams="a"/>
        <requestHandler name="/dump2" class="DumpRequestHandler" initParams="a">
          <lst name="defaults">
            <str name="a">A1</str>
          </lst>
          <lst name="invariants">
            <str name="b">B1</str>
          </lst>
          <lst name="appends">
            <str name="c">C1</str>
          </lst>
        </requestHandler>
      
      1. SOLR-6365.patch
        5 kB
        Noble Paul
      2. SOLR-6365.patch
        13 kB
        Noble Paul
      3. SOLR-6365.patch
        21 kB
        Noble Paul
      4. SOLR-6365-crappy-test.patch
        4 kB
        Erick Erickson

        Issue Links

          Activity

          Hide
          Hoss Man added a comment -

          <params path="/some-other-path*" defaults="a=b&c=d&e=f" invariants="x=y" appends="i=j"/>

          that's not even valid XML (bare &)

          and what does it even mean to say that you want to set some defaults and invariants on /some-other-path* if you don't configure any type of information about what handler /some-other-path* uses?

          how would this kind of syntax help with "...we can avoid specifying the components altogether and make solrconfig much simpler." ?

          Show
          Hoss Man added a comment - <params path="/some-other-path*" defaults="a=b&c=d&e=f" invariants="x=y" appends="i=j"/> that's not even valid XML (bare & ) and what does it even mean to say that you want to set some defaults and invariants on /some-other-path* if you don't configure any type of information about what handler /some-other-path* uses? how would this kind of syntax help with "...we can avoid specifying the components altogether and make solrconfig much simpler." ?
          Hide
          Erick Erickson added a comment -

          I'm a bit puzzled too at what the point is here. From a sysadmin's standpoint,
          this would move all the configuration (which is vitally important to me) to
          some scattered code that lives on, like, people's personal laptops, a nightmare
          to administer.

          So I guess you're thinking of some higher-level problem that this is part of,
          what is that problem? A REST API for solrconfig?

          Show
          Erick Erickson added a comment - I'm a bit puzzled too at what the point is here. From a sysadmin's standpoint, this would move all the configuration (which is vitally important to me) to some scattered code that lives on, like, people's personal laptops, a nightmare to administer. So I guess you're thinking of some higher-level problem that this is part of, what is that problem? A REST API for solrconfig?
          Hide
          Noble Paul added a comment - - edited

          that's not even valid XML (bare &)

          yeah, you are right, according to xml standards it is not. But all parsers accept that . But that is besides the point

          and what does it even mean to say that you want to set some defaults and invariants on /some-other-path/* if you don't configure any type of information about what handler /some-other-path/ uses?

          Yes, Looking from a user's point of view. They don't really think about the solr components. They assume that a given path , say /update, has certain capabilities and accepts certain parameters . For them it is not a component , it is just an API end point. Yes, you can of course specify wrong parameters which you are free to do even now. I'm not saying we will take away all configuration from solrconfig.xml . It is mainly for the fixed paths.

          The other use case this addresses is our REST APIs. It is managed completely outside of solrconfig.xml and there is no way to specify params .

          how would this kind of syntax help with "...we can avoid specifying the components altogether and make solrconfig much simpler." ?

          I'm thinking of fixing certain paths and avoiding certain common definitions in the xml file. We should make it fixed saying that certain paths and their parameters are fixed say /select , /update, /admin/* , /get etc. All I should be able to do is set params

          In the current design it is impossible to have global level configurations which spans multiple components , say wt=json or df=text for all paths.

          So I guess you're thinking of some higher-level problem that this is part of, what is that problem? A REST API for solrconfig?

          Yes, you are right , this issue is not addressing that use case, But it becomes much simpler to provide an API to modify params than the entire components. Most often the usecase is about changing the params

          Show
          Noble Paul added a comment - - edited that's not even valid XML (bare &) yeah, you are right, according to xml standards it is not. But all parsers accept that . But that is besides the point and what does it even mean to say that you want to set some defaults and invariants on /some-other-path/* if you don't configure any type of information about what handler /some-other-path/ uses? Yes, Looking from a user's point of view. They don't really think about the solr components. They assume that a given path , say /update , has certain capabilities and accepts certain parameters . For them it is not a component , it is just an API end point. Yes, you can of course specify wrong parameters which you are free to do even now. I'm not saying we will take away all configuration from solrconfig.xml . It is mainly for the fixed paths. The other use case this addresses is our REST APIs. It is managed completely outside of solrconfig.xml and there is no way to specify params . how would this kind of syntax help with "...we can avoid specifying the components altogether and make solrconfig much simpler." ? I'm thinking of fixing certain paths and avoiding certain common definitions in the xml file. We should make it fixed saying that certain paths and their parameters are fixed say /select , /update , /admin/* , /get etc. All I should be able to do is set params In the current design it is impossible to have global level configurations which spans multiple components , say wt=json or df=text for all paths. So I guess you're thinking of some higher-level problem that this is part of, what is that problem? A REST API for solrconfig? Yes, you are right , this issue is not addressing that use case, But it becomes much simpler to provide an API to modify params than the entire components. Most often the usecase is about changing the params
          Hide
          Shalin Shekhar Mangar added a comment -

          I like this idea. We can also provide a way to name a certain defaults/appends/invariants combination such that people can just provide a name while querying. This will become more powerful when we build REST APIs for creating/modifying such named param-sets.

          From a sysadmin's standpoint, this would move all the configuration (which is vitally important to me) to some scattered code that lives on, like, people's personal laptops, a nightmare to administer.

          I didn't get that impression from reading the description. What makes you say that?

          Show
          Shalin Shekhar Mangar added a comment - I like this idea. We can also provide a way to name a certain defaults/appends/invariants combination such that people can just provide a name while querying. This will become more powerful when we build REST APIs for creating/modifying such named param-sets. From a sysadmin's standpoint, this would move all the configuration (which is vitally important to me) to some scattered code that lives on, like, people's personal laptops, a nightmare to administer. I didn't get that impression from reading the description. What makes you say that?
          Hide
          Noble Paul added a comment -

          We can also provide a way to name a certain defaults/appends/invariants combination

          I like that idea, naming a bunch of params and using it as a reference in queries

          Show
          Noble Paul added a comment - We can also provide a way to name a certain defaults/appends/invariants combination I like that idea, naming a bunch of params and using it as a reference in queries
          Hide
          Erik Hatcher added a comment - - edited

          naming a bunch of params and using it as a reference in queries

          +1, and I'll add a bit of interesting historical correlation to Ant's "data types" http://ant.apache.org/manual/using.html#path

          I'd suggest rather than trying to make the params be represented as HTTP query string fragments (a messy implementation detail, embedded solr usage for example doesn't need to talk HTTP or query strings), that they be <lst name="defaults"><str name="param_name">param_value</str></lst> kind of format. In the spirit of the Ant, maybe something like:

            <paramset id="my_facet_params">
              <lst name="defaults">
                <str name="facet.field">category</str>
                <!-- ... -->
              </lst>
            </paramset>
          

          And then request handlers could pick up one or more parameter sets such as /select?q=query&paramset=my_facet_params (or maybe &paramsets=my_facet_params,.... so they can be in guaranteed order of evaluation).

          Show
          Erik Hatcher added a comment - - edited naming a bunch of params and using it as a reference in queries +1, and I'll add a bit of interesting historical correlation to Ant's "data types" http://ant.apache.org/manual/using.html#path I'd suggest rather than trying to make the params be represented as HTTP query string fragments (a messy implementation detail, embedded solr usage for example doesn't need to talk HTTP or query strings), that they be <lst name="defaults"><str name="param_name">param_value</str></lst> kind of format. In the spirit of the Ant, maybe something like: <paramset id= "my_facet_params" > <lst name= "defaults" > <str name= "facet.field" >category</str> <!-- ... --> </lst> </paramset> And then request handlers could pick up one or more parameter sets such as /select?q=query&paramset=my_facet_params (or maybe &paramsets=my_facet_params,.... so they can be in guaranteed order of evaluation).
          Hide
          Noble Paul added a comment - - edited

          There are clearly two aspects to this ticket.

          First and the important one is the ability to define params(or paramsets ) outside of components . Reference them by name in request/component config, or assign them to paths etc. I see params as just name-values used in requests or component initialization and not as integral parts of components. While, we the devs ,see Solr as a system of carefully assembled components as per an xml configuration (solrconfig.xml) , the average user sees Solr as a server which supports a bunch of APIs (of http paths) . I don't see a reason why a certain component should not be there as long as it is not consuming resources (lazy loading) or is a security hole (Tika?) . So , all the paths are available all the time and fixed (unless a user explicitly overrides it) . Once we separate out the params we have better ways to configure them via REST apis or in a zookeeper node .

          The second part is the syntax and we should move towards a syntax that is more palatable to the 'new generation' and not just the old timers.I'm sure we can arrive at a reasonably clean way to put them in solrconfig.xml and we should choose one that is best for our users . I see no reason to have type info (str,int,bool etc) in configuration when http has no ways to support them . So, all components will have to assume that all variables are strings and parse them accordingly .

          compare the following

          <!-- use json for all paths and _txt as the default search field-->
          <params id="global" path="/**" defaults="wt:'json',df:'_txt'"  />
          
          

          or

          <!-- use json for all paths and _txt as the default search field-->
          <params id="global" path="/**">
            <lst name="defaults">
               <str name="wt">json</str>
               <str name="df"_txt</str>
            </lst>
          </params>
          

          Please suggest other options if you can think of

          Show
          Noble Paul added a comment - - edited There are clearly two aspects to this ticket. First and the important one is the ability to define params(or paramsets ) outside of components . Reference them by name in request/component config, or assign them to paths etc. I see params as just name-values used in requests or component initialization and not as integral parts of components. While, we the devs ,see Solr as a system of carefully assembled components as per an xml configuration (solrconfig.xml) , the average user sees Solr as a server which supports a bunch of APIs (of http paths) . I don't see a reason why a certain component should not be there as long as it is not consuming resources (lazy loading) or is a security hole (Tika?) . So , all the paths are available all the time and fixed (unless a user explicitly overrides it) . Once we separate out the params we have better ways to configure them via REST apis or in a zookeeper node . The second part is the syntax and we should move towards a syntax that is more palatable to the 'new generation' and not just the old timers.I'm sure we can arrive at a reasonably clean way to put them in solrconfig.xml and we should choose one that is best for our users . I see no reason to have type info (str,int,bool etc) in configuration when http has no ways to support them . So, all components will have to assume that all variables are strings and parse them accordingly . compare the following <!-- use json for all paths and _txt as the default search field--> <params id= "global" path= "/**" defaults= "wt:'json',df:'_txt'" /> or <!-- use json for all paths and _txt as the default search field--> <params id= "global" path= "/**" > <lst name= "defaults" > <str name= "wt" > json </str> <str name= "df" _txt</str> </lst> </params> Please suggest other options if you can think of
          Hide
          Varun Thacker added a comment -

          The second part is the syntax and we should move towards a syntax that is more palatable to the 'new generation' and not just the old timers

          I like the new syntax for defaults, appends and invariants.

          First and the important one is the ability to define params(or paramsets ) outside of components . Reference them by name in request/component config, or assign them to paths etc.

          Where do I say that path=/search should be used for searching

          Show
          Varun Thacker added a comment - The second part is the syntax and we should move towards a syntax that is more palatable to the 'new generation' and not just the old timers I like the new syntax for defaults, appends and invariants. First and the important one is the ability to define params(or paramsets ) outside of components . Reference them by name in request/component config, or assign them to paths etc. Where do I say that path=/search should be used for searching
          Hide
          Noble Paul added a comment - - edited

          Where do I say that path=/search should be used for searching

          No, users don't say what is the there in the path /search . we should decide it for the users. But if the users insist on doing that they can do that.

          BTW you should look at SOLR-6191 and apply the patch and play with it . if you know any path and append /meta to that path and you will get the documentation on that path.

          for instance /meta will tell you all the registered paths at the root. And /select will be listed there . If you hit /select/meta it will describe the capabilities+ params for /select

          And planning to add more things such as /select/meta/stats should give you stats for /select

          The objective is to take the focus away from the configuration xmls for the normal user and make the http end point act as the API and documentation

          Show
          Noble Paul added a comment - - edited Where do I say that path=/search should be used for searching No, users don't say what is the there in the path /search . we should decide it for the users. But if the users insist on doing that they can do that. BTW you should look at SOLR-6191 and apply the patch and play with it . if you know any path and append /meta to that path and you will get the documentation on that path. for instance /meta will tell you all the registered paths at the root. And /select will be listed there . If you hit /select/meta it will describe the capabilities+ params for /select And planning to add more things such as /select/meta/stats should give you stats for /select The objective is to take the focus away from the configuration xmls for the normal user and make the http end point act as the API and documentation
          Hide
          Varun Thacker added a comment -

          Okay so you are saying we will have fixed paths defined in Solr - /select, /update, /get, /suggest, /spellcheck etc.

          Now all the user has to do is define his paramset. And then their API calls would look like -
          /select?q=foo&paramset=category-search
          Or
          /select/param-set-name?q=foo

          Also if we are making this change would it also make sense to rename /select to /search ?

          How does one plug in their custom component?

          BTW you should look at SOLR-6191 and apply the patch and play with it . if you know any path and append /meta to that path and you will get the documentation on that path.

          Sounds awesome

          Show
          Varun Thacker added a comment - Okay so you are saying we will have fixed paths defined in Solr - /select, /update, /get, /suggest, /spellcheck etc. Now all the user has to do is define his paramset. And then their API calls would look like - /select?q=foo&paramset=category-search Or /select/param-set-name?q=foo Also if we are making this change would it also make sense to rename /select to /search ? How does one plug in their custom component? BTW you should look at SOLR-6191 and apply the patch and play with it . if you know any path and append /meta to that path and you will get the documentation on that path. Sounds awesome
          Hide
          Noble Paul added a comment - - edited

          How does one plug in their custom component?

          We are not taking away the ability of user to define his own components. .He can still do that and everything will work just as fine. For standard components ,we will have them implicitly defined. But, if the user overrides /update in solrconfig.xml , that will take precedence

          Show
          Noble Paul added a comment - - edited How does one plug in their custom component? We are not taking away the ability of user to define his own components. .He can still do that and everything will work just as fine. For standard components ,we will have them implicitly defined. But, if the user overrides /update in solrconfig.xml , that will take precedence
          Hide
          Noble Paul added a comment - - edited

          I'm going with the legacy solr way of doing this

          <!-- use json for all paths and _txt as the default search field-->
          <paramSet name="global" path="/**">
            <lst name="defaults">
               <str name="wt">json</str>
               <str name="df">_txt</str>
            </lst>
          </paramSet>
          

          The feature is more important than the syntax itself

          Show
          Noble Paul added a comment - - edited I'm going with the legacy solr way of doing this <!-- use json for all paths and _txt as the default search field--> <paramSet name= "global" path= "/**" > <lst name= "defaults" > <str name= "wt" > json </str> <str name= "df" > _txt </str> </lst> </paramSet> The feature is more important than the syntax itself
          Hide
          Noble Paul added a comment -

          Fix with testcases. I plan to commit this soon

          Show
          Noble Paul added a comment - Fix with testcases. I plan to commit this soon
          Hide
          ASF subversion and git services added a comment -

          Commit 1621414 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1621414 ]

          SOLR-6365: specify appends, defaults, invariants outside of the request handler

          Show
          ASF subversion and git services added a comment - Commit 1621414 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1621414 ] SOLR-6365 : specify appends, defaults, invariants outside of the request handler
          Hide
          ASF subversion and git services added a comment -

          Commit 1621489 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1621489 ]

          SOLR-6365: specify appends, defaults, invariants outside of the request handler

          Show
          ASF subversion and git services added a comment - Commit 1621489 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1621489 ] SOLR-6365 : specify appends, defaults, invariants outside of the request handler
          Hide
          ASF subversion and git services added a comment -

          Commit 1622156 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1622156 ]

          SOLR-6365 multiparamset bug

          Show
          ASF subversion and git services added a comment - Commit 1622156 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1622156 ] SOLR-6365 multiparamset bug
          Hide
          ASF subversion and git services added a comment -

          Commit 1622168 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1622168 ]

          SOLR-6365 NPE bug

          Show
          ASF subversion and git services added a comment - Commit 1622168 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1622168 ] SOLR-6365 NPE bug
          Hide
          ASF subversion and git services added a comment -

          Commit 1622169 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1622169 ]

          SOLR-6365 paramset bug

          Show
          ASF subversion and git services added a comment - Commit 1622169 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1622169 ] SOLR-6365 paramset bug
          Hide
          ASF subversion and git services added a comment -

          Commit 1622384 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1622384 ]

          SOLR-6365 refactoring and cleanup

          Show
          ASF subversion and git services added a comment - Commit 1622384 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1622384 ] SOLR-6365 refactoring and cleanup
          Hide
          ASF subversion and git services added a comment -

          Commit 1622385 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1622385 ]

          SOLR-6365 refactoring and cleanup

          Show
          ASF subversion and git services added a comment - Commit 1622385 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1622385 ] SOLR-6365 refactoring and cleanup
          Hide
          ASF subversion and git services added a comment -

          Commit 1622387 from Noble Paul in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1622387 ]

          SOLR-6365 refactoring and cleanup

          Show
          ASF subversion and git services added a comment - Commit 1622387 from Noble Paul in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1622387 ] SOLR-6365 refactoring and cleanup
          Hide
          Shalin Shekhar Mangar added a comment -

          Sorry for being late with feedback but I really think this shouldn't be called "paramSet". This is basically about refactoring the initial configuration for request handlers out of their section. In future when do have real query/param templates then this name will come back to bite us. We should call it what it is such as initArgs or something similar.

          Show
          Shalin Shekhar Mangar added a comment - Sorry for being late with feedback but I really think this shouldn't be called "paramSet". This is basically about refactoring the initial configuration for request handlers out of their section. In future when do have real query/param templates then this name will come back to bite us. We should call it what it is such as initArgs or something similar.
          Hide
          Noble Paul added a comment - - edited

          yeah let's make it initArgs how about just initParams ?

          Show
          Noble Paul added a comment - - edited yeah let's make it initArgs how about just initParams ?
          Hide
          Noble Paul added a comment -

          changed the tag name from <paramSet> to <initParams>

          Show
          Noble Paul added a comment - changed the tag name from <paramSet> to <initParams>
          Hide
          ASF subversion and git services added a comment -

          Commit 1626045 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1626045 ]

          SOLR-6365 <paramSet> renamed to <initParams>

          Show
          ASF subversion and git services added a comment - Commit 1626045 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1626045 ] SOLR-6365 <paramSet> renamed to <initParams>
          Hide
          ASF subversion and git services added a comment -

          Commit 1626052 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1626052 ]

          SOLR-6365 <paramSet> renamed to <initParams>

          Show
          ASF subversion and git services added a comment - Commit 1626052 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1626052 ] SOLR-6365 <paramSet> renamed to <initParams>
          Hide
          Erick Erickson added a comment -

          Is this really unresolved at this point? Assuming so, I'll make the comments here but perhaps we should open a new JIRA instead.

          On a trunk build with the stock Solr example I see
          <initParams path="/update/**,/query,/select,/tvrh,/elevate,/spell,/browse">
          <lst name="defaults">
          <str name="df">text</str>
          </lst>
          </initParams>

          I changed the df param in my /select requestHandler to:
          <lst name="defaults">
          <str name="df">name</str>
          </lst>

          Then when I just issue ....collection1/select?q=whatever it parses to q=text:whatever rather than q=name:whatever, which is quite surprising.

          Is this the intended behavior? Or should defaults in the requestHandler override the initParams?

          Either way we need to do something. Either it's a bug in the current implementation and the defaults section of the individual handler should override the initParams or we should remove the defaults from all the request handlers in the sample solrconfig.xml. The current behavior is disconcerting to someone who hasn't followed this JIRA closely, i.e. almost all of our users.

          If we remove the defaults section from the request handlers in solrconfig.xml, I think it would be best to make an explicit reference to initParams, we need to give users some clue what's going on here. This assumes that the notion of being able to call out initParams by ID didn't fall by the wayside.

          But this will make the vexing problem with, say, people who remove the "text" field from schema.xml and then can't load cores soooo much easier to fix. Rather than finding all the places that the text field is referenced in solrconfig.xml and change them to something that is in the schema, there'll be just one place to change......

          I've attached a test case for the trunk illustrating. I labeled it totally-crappy because it should NOT be used verbatim, it's a miserable hack to illustrate. I changed solrconfig-minimal.xml and schema-tiny.xml and have NOT re-run all tests with those changes and I fully expect those changes will break something.

          Show
          Erick Erickson added a comment - Is this really unresolved at this point? Assuming so, I'll make the comments here but perhaps we should open a new JIRA instead. On a trunk build with the stock Solr example I see <initParams path="/update/**,/query,/select,/tvrh,/elevate,/spell,/browse"> <lst name="defaults"> <str name="df">text</str> </lst> </initParams> I changed the df param in my /select requestHandler to: <lst name="defaults"> <str name="df">name</str> </lst> Then when I just issue ....collection1/select?q=whatever it parses to q=text:whatever rather than q=name:whatever, which is quite surprising. Is this the intended behavior? Or should defaults in the requestHandler override the initParams? Either way we need to do something. Either it's a bug in the current implementation and the defaults section of the individual handler should override the initParams or we should remove the defaults from all the request handlers in the sample solrconfig.xml. The current behavior is disconcerting to someone who hasn't followed this JIRA closely, i.e. almost all of our users. If we remove the defaults section from the request handlers in solrconfig.xml, I think it would be best to make an explicit reference to initParams, we need to give users some clue what's going on here. This assumes that the notion of being able to call out initParams by ID didn't fall by the wayside. But this will make the vexing problem with, say, people who remove the "text" field from schema.xml and then can't load cores soooo much easier to fix. Rather than finding all the places that the text field is referenced in solrconfig.xml and change them to something that is in the schema, there'll be just one place to change...... I've attached a test case for the trunk illustrating. I labeled it totally-crappy because it should NOT be used verbatim, it's a miserable hack to illustrate. I changed solrconfig-minimal.xml and schema-tiny.xml and have NOT re-run all tests with those changes and I fully expect those changes will break something.
          Hide
          Noble Paul added a comment -

          Is this the intended behavior? Or should defaults in the requestHandler override the initParams?

          It is intended behavior, but is it desirable . We can of course change it

          Show
          Noble Paul added a comment - Is this the intended behavior? Or should defaults in the requestHandler override the initParams? It is intended behavior, but is it desirable . We can of course change it
          Hide
          Erick Erickson added a comment -

          I haven't thought about it too deeply. Off the top of my head, having locally-defined parameters override the initParams seems best. Say you want to have a different default search field for the /query handler and the /select handler, but want all the other params to remain the same. You could call out the initParams, but add a single param in the "defaults" section of one of the handlers. Not entirely sure it's a valid example, but you see where it's going.

          I don't feel super-strongly about either approach. I come down somewhat on the side of the individual request handlers being able to override the initParams on the basis that inevitably there'll be a questions like "I just want to change one thing, why do I have to define a whole new initParams node?".

          I'll argue though that we should make it harder to fall into the trap I just fell into; taking my experience with solrconfig.xml and thinking I knew what I was doing. We could accomplish this just by taking all the defaults, invariants etc out of the individual request handlers. This would at least cause some head-scratching, perhaps with a comment in solrconfig.xml directing them to the initParams (and possibly this JIRA or the Reference Guide?) to let people know where to look for all of the details. I flat guarantee that people will copy/paste their old defaults (& etc) sections from old solrconfig files to the new one and be puzzled if we don't direct their attention appropriately.

          Show
          Erick Erickson added a comment - I haven't thought about it too deeply. Off the top of my head, having locally-defined parameters override the initParams seems best. Say you want to have a different default search field for the /query handler and the /select handler, but want all the other params to remain the same. You could call out the initParams, but add a single param in the "defaults" section of one of the handlers. Not entirely sure it's a valid example, but you see where it's going. I don't feel super-strongly about either approach. I come down somewhat on the side of the individual request handlers being able to override the initParams on the basis that inevitably there'll be a questions like "I just want to change one thing, why do I have to define a whole new initParams node?". I'll argue though that we should make it harder to fall into the trap I just fell into; taking my experience with solrconfig.xml and thinking I knew what I was doing. We could accomplish this just by taking all the defaults, invariants etc out of the individual request handlers. This would at least cause some head-scratching, perhaps with a comment in solrconfig.xml directing them to the initParams (and possibly this JIRA or the Reference Guide?) to let people know where to look for all of the details. I flat guarantee that people will copy/paste their old defaults (& etc) sections from old solrconfig files to the new one and be puzzled if we don't direct their attention appropriately.
          Hide
          Noble Paul added a comment -

          request_params > initParams> params in handler

          this means the

          values are applied in that order for defaults and in the opposite order for invariants

          so if you used "invariants" in the request handler , it would have taken precedence over initParams

          We can go either way

          I haven't thought about it too deeply. Off the top of my head, having locally-defined parameters override the initParams seems best.

          Just keep one thing in mind that "invariants" is expected to work differently than "defaults"

          Show
          Noble Paul added a comment - request_params > initParams > params in handler this means the values are applied in that order for defaults and in the opposite order for invariants so if you used "invariants" in the request handler , it would have taken precedence over initParams We can go either way I haven't thought about it too deeply. Off the top of my head, having locally-defined parameters override the initParams seems best. Just keep one thing in mind that "invariants" is expected to work differently than "defaults"
          Hide
          Erick Erickson added a comment -

          Hmm, it seems more intuitive that the order be

          initParams are superseded by params_in_handler which are superseded by request_params

          Invariants:
          Right, then I should think that the last step in the above is omitted.
          initParams are superseded by params_in_handler
          and request_params are ignored.

          I guess my thinking is that initParams are the most general bucket, and users may have reasons
          for different handlers to override some of them and people issuing requests would have the
          last chance to change them....

          I urge that we apply them in the same order for both invariants and defaults, I'd much rather
          learn one rule than two, and I really don't want to explain it over and over an over again
          to confused users

          Show
          Erick Erickson added a comment - Hmm, it seems more intuitive that the order be initParams are superseded by params_in_handler which are superseded by request_params Invariants: Right, then I should think that the last step in the above is omitted. initParams are superseded by params_in_handler and request_params are ignored. I guess my thinking is that initParams are the most general bucket, and users may have reasons for different handlers to override some of them and people issuing requests would have the last chance to change them.... I urge that we apply them in the same order for both invariants and defaults, I'd much rather learn one rule than two, and I really don't want to explain it over and over an over again to confused users
          Hide
          Erik Hatcher added a comment -
          Hmm, it seems more intuitive that the order be
          
          initParams are superseded by params_in_handler which are superseded by request_params
          
          Invariants:
          Right, then I should think that the last step in the above is omitted.
          initParams are superseded by params_in_handler
          and request_params are ignored.
          

          +1, I'm exactly with Erick Erickson on this.

          Show
          Erik Hatcher added a comment - Hmm, it seems more intuitive that the order be initParams are superseded by params_in_handler which are superseded by request_params Invariants: Right, then I should think that the last step in the above is omitted. initParams are superseded by params_in_handler and request_params are ignored. +1, I'm exactly with Erick Erickson on this.
          Hide
          Noble Paul added a comment -

          agreed
          I shall change this right away

          Show
          Noble Paul added a comment - agreed I shall change this right away
          Hide
          Noble Paul added a comment -

          changes the order of applying params.
          the variables defined in the requesthandler will always override initParams

          Show
          Noble Paul added a comment - changes the order of applying params. the variables defined in the requesthandler will always override initParams
          Hide
          ASF subversion and git services added a comment -

          Commit 1636330 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1636330 ]

          SOLR-6365

          Show
          ASF subversion and git services added a comment - Commit 1636330 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1636330 ] SOLR-6365
          Hide
          ASF subversion and git services added a comment -

          Commit 1636331 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1636331 ]

          SOLR-6365

          Show
          ASF subversion and git services added a comment - Commit 1636331 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1636331 ] SOLR-6365
          Hide
          Noble Paul added a comment -

          done.
          The new behavior is very simple. Whatever is put inside the <requestHandle> takes precedence over <initParams>

          Show
          Noble Paul added a comment - done. The new behavior is very simple. Whatever is put inside the <requestHandle> takes precedence over <initParams>
          Hide
          Erick Erickson added a comment -

          Great!

          Show
          Erick Erickson added a comment - Great!
          Hide
          Erick Erickson added a comment -

          Noble Paul Can we close this JIRA?

          Show
          Erick Erickson added a comment - Noble Paul Can we close this JIRA?
          Hide
          David Smiley added a comment -

          Wow I'm late to this; I wish there was a way to signal particular JIRA issues as worthy of more eyeballs.

          This feature reminds me of my feature-request "request handler inheritance" – SOLR-3293. I understand the design goals of this new "initParams" feature and it's nice to see this accomplished. What I don't like about the newly added "initParams" feature is that it seems redundant with request handlers themselves, and thus creates inevitable confusion as to what takes precedence, and it's another parameter (not a big deal but something), and it's another "thing" (the initParams) vs easily-grok'able feature of another "thing" (requestHandlers) which already exist. With request handler inheritance, you could have /select and /mainSearch and /selectHighlight and /selectAutoSuggest or whatever. Instead of making the parameter set the configurable thing that is managed by ZooKeeper & a REST API, why not do this for request handlers?

          What I like about user-created requestHandlers is (a) Solr keeps stats on them separately, and (b) they are easily distinguished in the URL. Other things I like about them are achieved in this patch, like changing a parameter relevant for some searches but not for others.

          Show
          David Smiley added a comment - Wow I'm late to this; I wish there was a way to signal particular JIRA issues as worthy of more eyeballs. This feature reminds me of my feature-request "request handler inheritance" – SOLR-3293 . I understand the design goals of this new "initParams" feature and it's nice to see this accomplished. What I don't like about the newly added "initParams" feature is that it seems redundant with request handlers themselves, and thus creates inevitable confusion as to what takes precedence, and it's another parameter (not a big deal but something), and it's another "thing" (the initParams) vs easily-grok'able feature of another "thing" (requestHandlers) which already exist. With request handler inheritance, you could have /select and /mainSearch and /selectHighlight and /selectAutoSuggest or whatever. Instead of making the parameter set the configurable thing that is managed by ZooKeeper & a REST API, why not do this for request handlers? What I like about user-created requestHandlers is (a) Solr keeps stats on them separately, and (b) they are easily distinguished in the URL. Other things I like about them are achieved in this patch, like changing a parameter relevant for some searches but not for others.
          Hide
          ASF subversion and git services added a comment -

          Commit 1640594 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1640594 ]

          SOLR-6365 "implicit requesthandler (specifiedin code) takes lower precedence over <initParams>"

          Show
          ASF subversion and git services added a comment - Commit 1640594 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1640594 ] SOLR-6365 "implicit requesthandler (specifiedin code) takes lower precedence over <initParams>"
          Hide
          ASF subversion and git services added a comment -

          Commit 1640595 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1640595 ]

          SOLR-6365 implicit requesthandler (specified in code) takes lower precedence over <initParams>

          Show
          ASF subversion and git services added a comment - Commit 1640595 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1640595 ] SOLR-6365 implicit requesthandler (specified in code) takes lower precedence over <initParams>
          Hide
          Alexandre Rafalovitch added a comment -

          Question: What's the application order of multiple initParams sections?

          I see in the current Solr 5 build that we have two initParams section, both applicable to /update/**. They do not conflict now, but there must be an implicit order in which they are applied, just in case somebody decides to do a Venn diagram of settings.

          I think, previously, the order was non-deterministic, as different components were just plucked out in whatever order. So, this might be the first time we need to care about this.

          Show
          Alexandre Rafalovitch added a comment - Question: What's the application order of multiple initParams sections? I see in the current Solr 5 build that we have two initParams section, both applicable to /update/** . They do not conflict now, but there must be an implicit order in which they are applied, just in case somebody decides to do a Venn diagram of settings. I think, previously, the order was non-deterministic, as different components were just plucked out in whatever order. So, this might be the first time we need to care about this.
          Hide
          Noble Paul added a comment -

          They do not conflict now, but there must be an implicit order in which they are applied, just in case somebody decides to do a Venn diagram of settings.

          The order in which they are applied is in the order they are written in solrconfig.xml

          I see in the current Solr 5 build that we have two initParams section, both applicable to /update/**

          one is a set of parameters applied on /update/** and the other is /update/json/docs . So, for the path /update/json/docs both <initParams> are added up

          Show
          Noble Paul added a comment - They do not conflict now, but there must be an implicit order in which they are applied, just in case somebody decides to do a Venn diagram of settings. The order in which they are applied is in the order they are written in solrconfig.xml I see in the current Solr 5 build that we have two initParams section, both applicable to /update/** one is a set of parameters applied on /update/** and the other is /update/json/docs . So, for the path /update/json/docs both <initParams> are added up
          Hide
          Hoss Man added a comment -

          something isn't working right here.

          the techproducts example has the following...

            <initParams path="/update/**,/query,/select,/tvrh,/elevate,/spell,/browse">
              <lst name="defaults">
                <str name="df">text</str>
              </lst>
            </initParams>
          

          ...but if you try to use /elevate like this...

          http://localhost:8983/solr/techproducts/elevate?q=ipod&enableElevation=true

          ...you get 400 error...

          >> no field name specified in query and no default specified via 'df' param

          Show
          Hoss Man added a comment - something isn't working right here. the techproducts example has the following... <initParams path= "/update/**,/query,/select,/tvrh,/elevate,/spell,/browse" > <lst name= "defaults" > <str name= "df" >text</str> </lst> </initParams> ...but if you try to use /elevate like this... http://localhost:8983/solr/techproducts/elevate?q=ipod&enableElevation=true ...you get 400 error... >> no field name specified in query and no default specified via 'df' param
          Hide
          Noble Paul added a comment -

          curious, let me check this

          Show
          Noble Paul added a comment - curious, let me check this
          Hide
          ASF subversion and git services added a comment -

          Commit 1646401 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1646401 ]

          SOLR-6365 lazy loaded components did not have initParams applied

          Show
          ASF subversion and git services added a comment - Commit 1646401 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1646401 ] SOLR-6365 lazy loaded components did not have initParams applied
          Hide
          ASF subversion and git services added a comment -

          Commit 1646402 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1646402 ]

          SOLR-6365 lazy loaded components did not have initParams applied

          Show
          ASF subversion and git services added a comment - Commit 1646402 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1646402 ] SOLR-6365 lazy loaded components did not have initParams applied
          Hide
          Noble Paul added a comment -

          when startup="lazy" was specified , <initParams> were not applied

          Show
          Noble Paul added a comment - when startup="lazy" was specified , <initParams> were not applied
          Hide
          Alexandre Rafalovitch added a comment -

          The matchPath(String path, String name) implementation is broken. It will match "/update" request against "/update/json/docs" and will use the later's defaults for the former.

          The problem is with returning true when falling out of the loop when the name is a shorter prefix of path.

          I have noticed it in the debug log when I did an update request (on Solr 5) and saw the unexpected default:

          DEBUG - 2015-01-07 02:20:59.970; org.apache.solr.update.processor.LogUpdateProcessor; PRE_UPDATE add{,id=GB18030TEST} {{params({params(),defaults(mapUniqueKeyOnly=true&df=text&srcField=_src_)}),defaults(wt=xml)}}
          
          Show
          Alexandre Rafalovitch added a comment - The matchPath(String path, String name) implementation is broken. It will match "/update" request against "/update/json/docs" and will use the later's defaults for the former. The problem is with returning true when falling out of the loop when the name is a shorter prefix of path . I have noticed it in the debug log when I did an update request (on Solr 5) and saw the unexpected default: DEBUG - 2015-01-07 02:20:59.970; org.apache.solr.update.processor.LogUpdateProcessor; PRE_UPDATE add{,id=GB18030TEST} {{params({params(),defaults(mapUniqueKeyOnly=true&df=text&srcField=_src_)}),defaults(wt=xml)}}
          Hide
          ASF subversion and git services added a comment -

          Commit 1649996 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1649996 ]

          SOLR-6365 bug fix matching wrong name when it is a shorter prefix of path

          Show
          ASF subversion and git services added a comment - Commit 1649996 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1649996 ] SOLR-6365 bug fix matching wrong name when it is a shorter prefix of path
          Hide
          ASF subversion and git services added a comment -

          Commit 1649997 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1649997 ]

          SOLR-6365 bug fix matching wrong name when it is a shorter prefix of path

          Show
          ASF subversion and git services added a comment - Commit 1649997 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1649997 ] SOLR-6365 bug fix matching wrong name when it is a shorter prefix of path
          Hide
          Noble Paul added a comment -

          thanks Alexandre Rafalovitch . fixed

          Show
          Noble Paul added a comment - thanks Alexandre Rafalovitch . fixed
          Hide
          ASF subversion and git services added a comment -

          Commit 1650064 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1650064 ]

          SOLR-6365 last commit caused regression

          Show
          ASF subversion and git services added a comment - Commit 1650064 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1650064 ] SOLR-6365 last commit caused regression
          Hide
          ASF subversion and git services added a comment -

          Commit 1650065 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1650065 ]

          SOLR-6365 last commit caused regression

          Show
          ASF subversion and git services added a comment - Commit 1650065 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1650065 ] SOLR-6365 last commit caused regression
          Hide
          ASF subversion and git services added a comment -

          Commit 1650125 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1650125 ]

          SOLR-6365 path /update/** should match /update as well

          Show
          ASF subversion and git services added a comment - Commit 1650125 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1650125 ] SOLR-6365 path /update/** should match /update as well
          Hide
          ASF subversion and git services added a comment -

          Commit 1650126 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1650126 ]

          SOLR-6365 path /update/** should match /update as well

          Show
          ASF subversion and git services added a comment - Commit 1650126 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1650126 ] SOLR-6365 path /update/** should match /update as well
          Hide
          Martin Grotzke added a comment -

          Moving to solr 5, I'm trying to configure the "healthcheckFile" for the PingRequestHandler.

          I added

            <initParams path="/admin/ping">
              <str name="healthcheckFile">server-enabled.txt</str>
            </initParams>
          

          to solrconfig, unfortunately this did not do the trick. I had to configure the PingRequestHandler completely to get the healthcheckFile configured.

          My assumption is that only appends, defaults and invariants can be specified outside of the component, so what I'm experiencing is expected and not a bug or an issue on my side. Is that correct?

          Show
          Martin Grotzke added a comment - Moving to solr 5, I'm trying to configure the "healthcheckFile" for the PingRequestHandler. I added <initParams path= "/admin/ping" > <str name= "healthcheckFile" >server-enabled.txt</str> </initParams> to solrconfig, unfortunately this did not do the trick. I had to configure the PingRequestHandler completely to get the healthcheckFile configured. My assumption is that only appends, defaults and invariants can be specified outside of the component, so what I'm experiencing is expected and not a bug or an issue on my side. Is that correct?
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.
          Hide
          Noble Paul added a comment -

          Martin Grotzke Plz open a seprate ticket and I shall fix it in 5.1

          Show
          Noble Paul added a comment - Martin Grotzke Plz open a seprate ticket and I shall fix it in 5.1
          Hide
          Martin Grotzke added a comment -

          Noble Paul Sounds great, I submitted SOLR-7157

          Show
          Martin Grotzke added a comment - Noble Paul Sounds great, I submitted SOLR-7157

            People

            • Assignee:
              Noble Paul
              Reporter:
              Noble Paul
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development