Solr
  1. Solr
  2. SOLR-5208

Support for the setting of core.properties key/values at create-time on Collections API

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.4
    • Fix Version/s: 4.7, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      As discussed on e-mail thread "Sharing SolrCloud collection configs w/overrides" (http://search-lucene.com/m/MUWXu1DIsqY1&subj=Sharing+SolrCloud+collection+configs+w+overrides), Erick brought up a neat solution using HTTP params at create-time for the Collection API.

      Essentially, this request is for a functionality that allows the setting of variables (core.properties) on Collections API CREATE command.

      Erick's idea:

      "Maybe it's as simple as allowing more params for creation like
      collection.coreName where each param of the form collection.blah=blort
      gets an entry in the properties file blah=blort?..."

      1. SOLR-5208.patch
        5 kB
        Erick Erickson
      2. SOLR-5208.patch
        3 kB
        Erick Erickson

        Activity

        Hide
        Tim Vaillancourt added a comment -

        Please assign to Erick Erickson.

        Show
        Tim Vaillancourt added a comment - Please assign to Erick Erickson.
        Hide
        Erick Erickson added a comment -

        Hey, man! I'm in training to be a manager, I just suggest stuff rather than, you know, actually do anything <G>...

        I can drive it through, but it'll get done faster some kind person puts together a patch...

        I guess on equestion is whether overriding the collection.blah is a good idea or whether it should be a different name like collectionProperty.blah... I have no real preference either way.

        Show
        Erick Erickson added a comment - Hey, man! I'm in training to be a manager, I just suggest stuff rather than, you know, actually do anything <G>... I can drive it through, but it'll get done faster some kind person puts together a patch... I guess on equestion is whether overriding the collection.blah is a good idea or whether it should be a different name like collectionProperty.blah... I have no real preference either way.
        Hide
        Mark Miller added a comment -

        The CoreAdmin api already supports this with the property.* param naming scheme on create calls - allowing this for the collections api is probably as simple as propagating any property.* params from the collections api call to the solrcore api subcalls. That seems like the best way to deal with this use case - the core properties should be persisted already, so very simple to add I think.

        Show
        Mark Miller added a comment - The CoreAdmin api already supports this with the property.* param naming scheme on create calls - allowing this for the collections api is probably as simple as propagating any property.* params from the collections api call to the solrcore api subcalls. That seems like the best way to deal with this use case - the core properties should be persisted already, so very simple to add I think.
        Hide
        Erick Erickson added a comment -

        Here's a Proof-of-Concept patch. It creates a core.properties file that has any property.key=value pair specified on the collection.create line reproduced as key=value. Mark was spot-on that it's just a matter of passing the params through to core creation..

        Mark Miller is that what you had in mind?

        Several questions though.

        1> Is copying the property.key=value necessary in CollectionsHandler.handleCreateShard?

        2> Similarly, should the property.key=value stuff be done in OverseerCollectionProcessor.createShard? What about splitShard? Just going by all the params.set that "look kinda like create" it seems possible at least.

        Show
        Erick Erickson added a comment - Here's a Proof-of-Concept patch. It creates a core.properties file that has any property.key=value pair specified on the collection.create line reproduced as key=value. Mark was spot-on that it's just a matter of passing the params through to core creation.. Mark Miller is that what you had in mind? Several questions though. 1> Is copying the property.key=value necessary in CollectionsHandler.handleCreateShard? 2> Similarly, should the property.key=value stuff be done in OverseerCollectionProcessor.createShard? What about splitShard? Just going by all the params.set that "look kinda like create" it seems possible at least.
        Hide
        Shalin Shekhar Mangar added a comment -

        +1 for adding this to splitshard and createshard as well.

        Show
        Shalin Shekhar Mangar added a comment - +1 for adding this to splitshard and createshard as well.
        Hide
        Tim Vaillancourt added a comment - - edited

        Thanks for the patch Erick!

        Shalin: good point on the SPLITSHARD. To be consistent, are there any other places this is needed?

        • Core API (already there).
        • Collections API "CREATE": discussed here.
        • Collections API "SPLITSHARD": Thanks Shalin!.
        • Collections API "CREATEALIAS": An alias shouldn't have it's own properties AFAIK, but calling that out.
        • Collections API "RELOAD": I'm not sure if the Core API functionality does this, but adding this to RELOAD would allow changing of properties post-create-time. Without this you'd need to DELETE/CREATE to change properties, or bypass.

        Tim

        Show
        Tim Vaillancourt added a comment - - edited Thanks for the patch Erick! Shalin: good point on the SPLITSHARD. To be consistent, are there any other places this is needed? Core API (already there). Collections API "CREATE": discussed here. Collections API "SPLITSHARD": Thanks Shalin!. Collections API "CREATEALIAS" : An alias shouldn't have it's own properties AFAIK, but calling that out. Collections API "RELOAD" : I'm not sure if the Core API functionality does this, but adding this to RELOAD would allow changing of properties post-create-time. Without this you'd need to DELETE/CREATE to change properties, or bypass. Tim
        Hide
        Erick Erickson added a comment -

        Tim:

        Use At Your Own Risk!!! I just did a very quick test on it, didn't even write any tests yet. If you're brave it'd be great to see if it fixes your problem.

        About the other commands....

        SPLITSHARD - I expect so, it has to create another core so we have to at least copy the properties file over (which I'm not sure we do either).

        CREATALIAS - I doubt it. This shouldn't affect the core.properties.

        RELOAD - That's interesting, hadn't really thought about that. It seems possible to shoot yourself in the foot here though. I'm also not sure that the reload writes out the core.properties file already or not.

        Show
        Erick Erickson added a comment - Tim: Use At Your Own Risk!!! I just did a very quick test on it, didn't even write any tests yet. If you're brave it'd be great to see if it fixes your problem. About the other commands.... SPLITSHARD - I expect so, it has to create another core so we have to at least copy the properties file over (which I'm not sure we do either). CREATALIAS - I doubt it. This shouldn't affect the core.properties. RELOAD - That's interesting, hadn't really thought about that. It seems possible to shoot yourself in the foot here though. I'm also not sure that the reload writes out the core.properties file already or not.
        Hide
        Tim Vaillancourt added a comment -

        Thanks Erick.

        I agree on RELOAD - I'm not sure if that makes sense or not either, but thought of it randomly while listing those commands . I'll make a new JIRA to discuss if that is a good idea or not.

        Tim

        Show
        Tim Vaillancourt added a comment - Thanks Erick. I agree on RELOAD - I'm not sure if that makes sense or not either, but thought of it randomly while listing those commands . I'll make a new JIRA to discuss if that is a good idea or not. Tim
        Hide
        Erick Erickson added a comment -

        Man, I simply cannot make a test even start to work for this, and I'm past my point of being willing to spend more time on it. Every time I try to add property.key=value to the parameters, the persisted core.properties files do not have anything there. And the code never goes through the usual persist logic.

        But outside the test environment, I've manually tested both create and split and they work fine. So I'm going to commit this after I run through the precommit and test process and leave it to someone more ambitious to untangle the whole test bits.

        Show
        Erick Erickson added a comment - Man, I simply cannot make a test even start to work for this, and I'm past my point of being willing to spend more time on it. Every time I try to add property.key=value to the parameters, the persisted core.properties files do not have anything there. And the code never goes through the usual persist logic. But outside the test environment, I've manually tested both create and split and they work fine. So I'm going to commit this after I run through the precommit and test process and leave it to someone more ambitious to untangle the whole test bits.
        Hide
        ASF subversion and git services added a comment -

        Commit 1543969 from Erick Erickson in branch 'dev/trunk'
        [ https://svn.apache.org/r1543969 ]

        SOLR-5208: Support for the setting of core.properties key/values at create-time on Collections API

        Show
        ASF subversion and git services added a comment - Commit 1543969 from Erick Erickson in branch 'dev/trunk' [ https://svn.apache.org/r1543969 ] SOLR-5208 : Support for the setting of core.properties key/values at create-time on Collections API
        Hide
        Tim Vaillancourt added a comment -

        Thanks Erick! Will try to test. I owe you a patch once I learn Java .

        Show
        Tim Vaillancourt added a comment - Thanks Erick! Will try to test. I owe you a patch once I learn Java .
        Hide
        Erick Erickson added a comment -

        Tim:

        Haven't checked it into 4x quite yet, will in another hour or so...

        You could always help Stefan out with the UI if that's in your skill set .

        Show
        Erick Erickson added a comment - Tim: Haven't checked it into 4x quite yet, will in another hour or so... You could always help Stefan out with the UI if that's in your skill set .
        Hide
        ASF subversion and git services added a comment -

        Commit 1543996 from Erick Erickson in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1543996 ]

        SOLR-5208: Support for the setting of core.properties key/values at create-time on Collections API

        Show
        ASF subversion and git services added a comment - Commit 1543996 from Erick Erickson in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1543996 ] SOLR-5208 : Support for the setting of core.properties key/values at create-time on Collections API
        Hide
        David Smiley added a comment -

        Erick, your condition checks the parameter as follows:

         param.indexOf(OverseerCollectionProcessor.COLL_PROP_PREFIX) != -1 

        Shouldn't you simply use String.startsWith versus checking at any arbitrary offset?

        Show
        David Smiley added a comment - Erick, your condition checks the parameter as follows: param.indexOf(OverseerCollectionProcessor.COLL_PROP_PREFIX) != -1 Shouldn't you simply use String.startsWith versus checking at any arbitrary offset?
        Hide
        David Smiley added a comment -

        Oh here's a question – how do you change these properties after the collection is created?

        Show
        David Smiley added a comment - Oh here's a question – how do you change these properties after the collection is created?
        Hide
        Erick Erickson added a comment -

        David:

        bq: Shouldn't you simply use String.startsWith versus checking at any arbitrary offset?

        Yep, this is one of those things that would come out in a very weird place sometime long after I'd completely forgotten about this. Thanks! See: SOLR-5491

        bq: Oh here's a question – how do you change these properties after the collection is created?

        A text editor and Puppet . I suppose one could build something in to, say, reload. I don't have a compelling use case to warrant the effort though so I'm not volunteering...

        Show
        Erick Erickson added a comment - David: bq: Shouldn't you simply use String.startsWith versus checking at any arbitrary offset? Yep, this is one of those things that would come out in a very weird place sometime long after I'd completely forgotten about this. Thanks! See: SOLR-5491 bq: Oh here's a question – how do you change these properties after the collection is created? A text editor and Puppet . I suppose one could build something in to, say, reload. I don't have a compelling use case to warrant the effort though so I'm not volunteering...
        Hide
        Tim Vaillancourt added a comment -

        "A text editor and Puppet" yep, lol.

        David: I opened up that can-of-worms over in this JIRA SOLR-5225. Any votes/ideas/input appreciated!

        Show
        Tim Vaillancourt added a comment - "A text editor and Puppet" yep, lol. David: I opened up that can-of-worms over in this JIRA SOLR-5225 . Any votes/ideas/input appreciated!

          People

          • Assignee:
            Erick Erickson
            Reporter:
            Tim Vaillancourt
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development