Solr
  1. Solr
  2. SOLR-4662

Finalize what we're going to do with solr.xml, auto-discovery, config sets.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.3, 6.0
    • Fix Version/s: 4.3, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Spinoff from SOLR-4615, breaking it out here so we can address the changes in pieces.

      1. SOLR-4662.patch
        31 kB
        Erick Erickson
      2. SOLR-4662.patch
        30 kB
        Mark Miller
      3. SOLR-4662.patch
        81 kB
        Erick Erickson
      4. SOLR-4662.patch
        75 kB
        Erick Erickson
      5. SOLR-4662.patch
        74 kB
        Erick Erickson
      6. SOLR-4662.patch
        55 kB
        Erick Erickson

        Issue Links

          Activity

          Hide
          Erick Erickson added a comment -

          I may actually have some time on some long flights to work on this, I'd really like to nail this and get it over. So I'd like to get the design agreed so I don't have to re-do the re-done.

          It seems like the root issue here is that we have this configuration file that is trying to serve too many purposes. It started as a way to get basic configuration information back in the single-core days. Then it morphed into a place to get all the core information. Then it morphed into a place where we could specify a bunch of SolrCloud parameters. Then it morphed into a place to not really get core information but specify that that info get discovered. Now, with SolrCloud one question is whether we should even have it or should all this info be stored in Zookeeper. Having local files around that you have to have on each node before starting solr seems counter to having configurations managed by ZK.

          Straw-man proposal:

          So, what if we allow a "SolrConfigurationProvider"? Specifying none or the default we'll provide does just what happens now. It seems like the other stock one we could supply is one that reads the info from ZK. Don't know what else is out there, but that's just the point. People could plug in something that read their DB, something that read a properties file <G>. Whatever. There's a start on what the interface would look like here in ConfigSolr, which I pulled out when trying to make a properties file. I doubt it's entirely complete, but should be a pretty good start. Maybe this becomes an option on the command line? e.g. -Dsolr.configProvider="org.solr.core.XmlConfigProvider" as the default? And if -DzkHost="url to zk" is provided, default to -Dsolr.configProvider="org.solr.core.ZkConfigProvider"?

          It seems like default behavior is then mostly a matter of which provider we use if none is specified. It'd be a pretty simple matter to switch between whether we use ZK or a solr.xml file based on whether we had a zkHost specified, and prefer the one that reads solr.xml in 4.x and the one that reads ZK in 5.x or whatever...

          Let me be the first to say that I have NOT thought this through very carefully, and note how I'm glossing over getting the appropriate info up into ZK in the first place. My purpose here is to get the discussion going, I don't particularly care if this approach is adopted or not. I do care that whatever we decide on doesn't do another 180 turn...

          Let the discussion begin!

          Show
          Erick Erickson added a comment - I may actually have some time on some long flights to work on this, I'd really like to nail this and get it over . So I'd like to get the design agreed so I don't have to re-do the re-done. It seems like the root issue here is that we have this configuration file that is trying to serve too many purposes. It started as a way to get basic configuration information back in the single-core days. Then it morphed into a place to get all the core information. Then it morphed into a place where we could specify a bunch of SolrCloud parameters. Then it morphed into a place to not really get core information but specify that that info get discovered. Now, with SolrCloud one question is whether we should even have it or should all this info be stored in Zookeeper. Having local files around that you have to have on each node before starting solr seems counter to having configurations managed by ZK. Straw-man proposal: So, what if we allow a "SolrConfigurationProvider"? Specifying none or the default we'll provide does just what happens now. It seems like the other stock one we could supply is one that reads the info from ZK. Don't know what else is out there, but that's just the point. People could plug in something that read their DB, something that read a properties file <G>. Whatever. There's a start on what the interface would look like here in ConfigSolr, which I pulled out when trying to make a properties file. I doubt it's entirely complete, but should be a pretty good start. Maybe this becomes an option on the command line? e.g. -Dsolr.configProvider="org.solr.core.XmlConfigProvider" as the default? And if -DzkHost="url to zk" is provided, default to -Dsolr.configProvider="org.solr.core.ZkConfigProvider"? It seems like default behavior is then mostly a matter of which provider we use if none is specified. It'd be a pretty simple matter to switch between whether we use ZK or a solr.xml file based on whether we had a zkHost specified, and prefer the one that reads solr.xml in 4.x and the one that reads ZK in 5.x or whatever... Let me be the first to say that I have NOT thought this through very carefully, and note how I'm glossing over getting the appropriate info up into ZK in the first place. My purpose here is to get the discussion going, I don't particularly care if this approach is adopted or not. I do care that whatever we decide on doesn't do another 180 turn... Let the discussion begin!
          Hide
          Mark Miller added a comment -

          Quick notes:

          I don't think it has really morphed.

          solr.xml has always been our only config file for the system or corecontainer level. Everything that has joined the corecontainer was added because it's not core level, and that's always been the role of solr.xml - home of the above SolrCore level.

          Let's talk about the overall strategy before we nail down the impl.

          IMO, solr.xml has a few issues:

          1. It should not define cores. Solr should not have to write to this config file, and cores should be define by directory locations.
          2. The SolrCloud params are all cores attributes, but they should be in a better structure, similar to the shard handler definition that can now live in solr.xml, and what we do in solrconfig.xml.
          3. We should consider how most of the solr.xml config can live in ZooKeeper when in SolrCloud mode. Much of this is not something you want to have to tweak on every node - it's going to be the same - you generally want to set a zkClientTimeout for your collection, not for each node. Same for shard handler. I think it's worth forcing some consistency for all nodes, such as the same host context. It will also be possible to override those things per node with sys prop notation. zkHost could no longer be specified in solr.xml.

          So, I think we need to be able to detect and read the old solr.xml format some way (lot's of choices I think), as well as detect and read the new format (which will be very similar to the old). In 5 we stop supporting the old format. Example ships with the new format.

          In terms of impl, I think you are on the right track. I don't think we should expose any user plugins at this point though. That should be internal work. Let's decide if we should support arbitrary impls for users later. It has an ongoing cost, in this area in particular I think. I think we want to reserve all API rights here.

          Show
          Mark Miller added a comment - Quick notes: I don't think it has really morphed. solr.xml has always been our only config file for the system or corecontainer level. Everything that has joined the corecontainer was added because it's not core level, and that's always been the role of solr.xml - home of the above SolrCore level. Let's talk about the overall strategy before we nail down the impl. IMO, solr.xml has a few issues: 1. It should not define cores. Solr should not have to write to this config file, and cores should be define by directory locations. 2. The SolrCloud params are all cores attributes, but they should be in a better structure, similar to the shard handler definition that can now live in solr.xml, and what we do in solrconfig.xml. 3. We should consider how most of the solr.xml config can live in ZooKeeper when in SolrCloud mode. Much of this is not something you want to have to tweak on every node - it's going to be the same - you generally want to set a zkClientTimeout for your collection, not for each node. Same for shard handler. I think it's worth forcing some consistency for all nodes, such as the same host context. It will also be possible to override those things per node with sys prop notation. zkHost could no longer be specified in solr.xml. So, I think we need to be able to detect and read the old solr.xml format some way (lot's of choices I think), as well as detect and read the new format (which will be very similar to the old). In 5 we stop supporting the old format. Example ships with the new format. In terms of impl, I think you are on the right track. I don't think we should expose any user plugins at this point though. That should be internal work. Let's decide if we should support arbitrary impls for users later. It has an ongoing cost, in this area in particular I think. I think we want to reserve all API rights here.
          Hide
          Mark Miller added a comment -

          So from my point of view:

          1. is essentially done
          2. is what we should tie up here, keeping in mind 3
          3. we should make another jira issue and follow on with

          Show
          Mark Miller added a comment - So from my point of view: 1. is essentially done 2. is what we should tie up here, keeping in mind 3 3. we should make another jira issue and follow on with
          Hide
          Shawn Heisey added a comment -

          I too think that solr.xml should not store cores. Auto-discovery is a nice thing, but it makes it more difficult to have a nice clean layout where the core configs and data are contained within separate subdirectories. I suppose that each autodiscovered core properties file could have a relative dataDir that goes up one or two directories, just like my current setup, and that might be good enough. I would also want to have something like a coreDirectory parameter, which I would set to "cores" so that the main solr.solr.home directory would be very clean.

          One option is to have a cores.xml file for more complex layouts where the user wants to specify everything. At first I thought about a cloud.xml file for SolrCloud properties, but once you start down the path of multiple config files, the explosion might never stop.

          Here's what things look like inside solr.xml on my setup that's not well suited for SolrCloud, and a snippet of the full directory structure. The full picture is more complex than this, with relative symlinks in conf directories and relative xinclude paths in solrconfig.xml, but this gives you the basic idea:

              <core loadOnStartup="true" instanceDir="cores/inc_0/" transient="false" name="inclive" dataDir="../../data/inc_0"/>
              <core loadOnStartup="true" instanceDir="cores/inc_1/" transient="false" name="incbuild" dataDir="../../data/inc_1"/>
          
          ├── cores
          │   ├── inc_0
          │   │   └── conf
          │   ├── inc_1
          │   │   └── conf
          ├── data
          │   ├── inc_0
          │   │   ├── index
          │   │   └── tlog
          │   ├── inc_1
          │   │   ├── index
          │   │   └── tlog
          └── lib
          

          I'm not sure there is a better location than solr.xml for zkHost, but I definitely think it should be in a config file. *Requiring* -DzkHost on startup is a bad idea - generic startup scripts become very difficult. Keeping all of the config after that in zookeeper is an idea that is growing on me, as long as the initial contact to zookeeper won't suffer from a low default client timeout before zkClientTimeout can be read.

          If you give Solr (or CloudSolrServer) only one of the hosts in your zookeeper ensemble (perhaps localhost:port/chroot) will it automatically configure itself with the entire ensemble? If not, is there a way to build this functionality in?

          The fact that Solr does not come equipped with instructions for a standalone zookeeper process or a functioning standalone zookeeper is problematic in my opinion. If I find some time, I can whip up a wiki page. I do wish zkcli.sh was more self-contained. I can eventually come up with a wiki page for that too.

          Show
          Shawn Heisey added a comment - I too think that solr.xml should not store cores. Auto-discovery is a nice thing, but it makes it more difficult to have a nice clean layout where the core configs and data are contained within separate subdirectories. I suppose that each autodiscovered core properties file could have a relative dataDir that goes up one or two directories, just like my current setup, and that might be good enough. I would also want to have something like a coreDirectory parameter, which I would set to "cores" so that the main solr.solr.home directory would be very clean. One option is to have a cores.xml file for more complex layouts where the user wants to specify everything. At first I thought about a cloud.xml file for SolrCloud properties, but once you start down the path of multiple config files, the explosion might never stop. Here's what things look like inside solr.xml on my setup that's not well suited for SolrCloud, and a snippet of the full directory structure. The full picture is more complex than this, with relative symlinks in conf directories and relative xinclude paths in solrconfig.xml, but this gives you the basic idea: <core loadOnStartup="true" instanceDir="cores/inc_0/" transient="false" name="inclive" dataDir="../../data/inc_0"/> <core loadOnStartup="true" instanceDir="cores/inc_1/" transient="false" name="incbuild" dataDir="../../data/inc_1"/> ├── cores │ ├── inc_0 │ │ └── conf │ ├── inc_1 │ │ └── conf ├── data │ ├── inc_0 │ │ ├── index │ │ └── tlog │ ├── inc_1 │ │ ├── index │ │ └── tlog └── lib I'm not sure there is a better location than solr.xml for zkHost, but I definitely think it should be in a config file. * Requiring * -DzkHost on startup is a bad idea - generic startup scripts become very difficult. Keeping all of the config after that in zookeeper is an idea that is growing on me, as long as the initial contact to zookeeper won't suffer from a low default client timeout before zkClientTimeout can be read. If you give Solr (or CloudSolrServer) only one of the hosts in your zookeeper ensemble (perhaps localhost:port/chroot) will it automatically configure itself with the entire ensemble? If not, is there a way to build this functionality in? The fact that Solr does not come equipped with instructions for a standalone zookeeper process or a functioning standalone zookeeper is problematic in my opinion. If I find some time, I can whip up a wiki page. I do wish zkcli.sh was more self-contained. I can eventually come up with a wiki page for that too.
          Hide
          Erick Erickson added a comment -

          Hmmmm. Point taken about not exposing a pluggable version yet, now that you mention it that seems premature. Hardening all that before exposing something we then have to support makes sense. We can still program to the interface I mentioned as much as possible to make that easier in future but no big deal either way.

          <1> is done.
          <2> OK, what does that look like? Is this just a simple transformation like
          <solr>
          <solrConfigFactory name="solrConfigFactory" clase="SolrConfigFactory">
          <str name="persistent">$

          {solr.xml.persist:false}

          </str>
          <str name="adminPath">/admin/cores</str>
          <str name="defaultCoreName">collection1</str>
          <str name="host">127.0.0.1</str>
          <str name="hostPort">$

          {hostPort:8983}

          </str>
          <str name="hostContext">$

          {hostContext:solr}

          </str>
          <str name="shareSchema">$

          {shareSchema:false}

          </str>

          <shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory">
          <int name="socketTimeout">$

          {socketTimeout:120000}

          </int>
          <int name="connTimeout">$

          {connTimeout:15000}

          </int>
          </shardHandlerFactory>

          <str name="zkClientTimeout">$

          {solr.zkclienttimeout:30000}

          </str>
          <str name="numShards">$

          {numShards:3}

          </str>
          <str name="distribUpdateConnTimeout">$

          {distribUpdateConnTimeout:15000}

          </str>
          <str name="distribUpdateSoTimeout">$

          {distribUpdateSoTimeout:120000}

          </str>
          </solrConfigFactory>

          </solr>

          This form doesn't allow cores to be here at all. Is there any necessity to have a factory here or are you thinking these should just be child nodes of <solr>? Would shardHandlerFactory be inside or outside the new factory? As you can tell, how all this fits together is something of a mystery to me. But having a solrConfigFactory node as the immediate child of <solr> and encompassing all the rest would allow easy detection of old vs new style XML. Adding a version attribute to the <solr> tag seems possible too, but I don't really like that, I think there's less user-level maintenance if we use solrConfigFactory, implementing up different handlers for different versions if/when we change again, which should be very rare.

          <3> On that note, it's not clear to me we need solr.xml at all in "true cloud mode". In the absence of solr.xml but presence of zkHost, just get everything from ZK. Look, you know how ZK-ignorant I am, be gentle <G>. The rest of the time, anything you could possibly want from solr.xml that wasn't present, ask ZK for it and use defaults if neither source had it. By "not present", that means neither in the solr.xml file nor as a sysprop. Along the way removing DEF_SOLR_XML from ConfigSolrXml would be a Good Thing maybe.

          Show
          Erick Erickson added a comment - Hmmmm. Point taken about not exposing a pluggable version yet, now that you mention it that seems premature. Hardening all that before exposing something we then have to support makes sense. We can still program to the interface I mentioned as much as possible to make that easier in future but no big deal either way. <1> is done. <2> OK, what does that look like? Is this just a simple transformation like <solr> <solrConfigFactory name="solrConfigFactory" clase="SolrConfigFactory"> <str name="persistent">$ {solr.xml.persist:false} </str> <str name="adminPath">/admin/cores</str> <str name="defaultCoreName">collection1</str> <str name="host">127.0.0.1</str> <str name="hostPort">$ {hostPort:8983} </str> <str name="hostContext">$ {hostContext:solr} </str> <str name="shareSchema">$ {shareSchema:false} </str> <shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory"> <int name="socketTimeout">$ {socketTimeout:120000} </int> <int name="connTimeout">$ {connTimeout:15000} </int> </shardHandlerFactory> <str name="zkClientTimeout">$ {solr.zkclienttimeout:30000} </str> <str name="numShards">$ {numShards:3} </str> <str name="distribUpdateConnTimeout">$ {distribUpdateConnTimeout:15000} </str> <str name="distribUpdateSoTimeout">$ {distribUpdateSoTimeout:120000} </str> </solrConfigFactory> </solr> This form doesn't allow cores to be here at all. Is there any necessity to have a factory here or are you thinking these should just be child nodes of <solr>? Would shardHandlerFactory be inside or outside the new factory? As you can tell, how all this fits together is something of a mystery to me. But having a solrConfigFactory node as the immediate child of <solr> and encompassing all the rest would allow easy detection of old vs new style XML. Adding a version attribute to the <solr> tag seems possible too, but I don't really like that, I think there's less user-level maintenance if we use solrConfigFactory, implementing up different handlers for different versions if/when we change again, which should be very rare. <3> On that note, it's not clear to me we need solr.xml at all in "true cloud mode". In the absence of solr.xml but presence of zkHost, just get everything from ZK. Look, you know how ZK-ignorant I am, be gentle <G>. The rest of the time, anything you could possibly want from solr.xml that wasn't present, ask ZK for it and use defaults if neither source had it. By "not present", that means neither in the solr.xml file nor as a sysprop. Along the way removing DEF_SOLR_XML from ConfigSolrXml would be a Good Thing maybe.
          Hide
          Erick Erickson added a comment -

          Shawn:

          Couple of things:

          1> there's no reason you can't have your individual data dirs specified in core.properties point wherever you want. Not sure that handles your case but thought I'd mention it. I'm not keen on maintaining a separate "specify everything" xml file, does the ability to specify your data dir for each core do what you need?

          2> SOLR-4478 (another thing I hope to have done Real Soon now) will allow "configsets". Basically in <solr_home>/configs (TBD) there will be directories like "conf1", "conf2" etc, which will be full Solr configuration directories. Then you can specify a "configSet" property in your individual core.properties file and that core will make use of the named config set, incidentally sharing all the setup between cores that use the same config set (i.e. won't parse/load the solrconfig.xml and schema.xml separately for each core that names the same config set).

          Keep the comments coming!

          Show
          Erick Erickson added a comment - Shawn: Couple of things: 1> there's no reason you can't have your individual data dirs specified in core.properties point wherever you want. Not sure that handles your case but thought I'd mention it. I'm not keen on maintaining a separate "specify everything" xml file, does the ability to specify your data dir for each core do what you need? 2> SOLR-4478 (another thing I hope to have done Real Soon now) will allow "configsets". Basically in <solr_home>/configs (TBD) there will be directories like "conf1", "conf2" etc, which will be full Solr configuration directories. Then you can specify a "configSet" property in your individual core.properties file and that core will make use of the named config set, incidentally sharing all the setup between cores that use the same config set (i.e. won't parse/load the solrconfig.xml and schema.xml separately for each core that names the same config set). Keep the comments coming!
          Hide
          Shawn Heisey added a comment -

          does the ability to specify your data dir for each core do what you need?

          As long the dataDir specified in a core properties file retains all its current capabilities relative to instanceDir, then yes, that will work nicely. As I mentioned, I would like to see a config option (probably in solr.xml, name suggestion coreDirectory) to specify a relative or absolute path where it will look for auto-detected instanceDirs. For the "thousands of cores" users, this will be very important, so that solr.solr.home isn't cluttered - it might have solr.xml (or its replacement) and a handful of directories like cores, lib, and maybe data. I'd like to see that parameter default to "cores" but I am not opposed to it being "." or having no default value at all.

          Basically in <solr_home>/configs (TBD) there will be directories like "conf1", "conf2" etc.

          I like this idea. I would want to make sure that relative xincludes are possible, just as they currently are with config sets in zookeeper. If the xml goes away and gets replaced by a properties file, including subconfigs won't be as important, but it would be nice if it were still possible.

          Show
          Shawn Heisey added a comment - does the ability to specify your data dir for each core do what you need? As long the dataDir specified in a core properties file retains all its current capabilities relative to instanceDir, then yes, that will work nicely. As I mentioned, I would like to see a config option (probably in solr.xml, name suggestion coreDirectory) to specify a relative or absolute path where it will look for auto-detected instanceDirs. For the "thousands of cores" users, this will be very important, so that solr.solr.home isn't cluttered - it might have solr.xml (or its replacement) and a handful of directories like cores, lib, and maybe data. I'd like to see that parameter default to "cores" but I am not opposed to it being "." or having no default value at all. Basically in <solr_home>/configs (TBD) there will be directories like "conf1", "conf2" etc. I like this idea. I would want to make sure that relative xincludes are possible, just as they currently are with config sets in zookeeper. If the xml goes away and gets replaced by a properties file, including subconfigs won't be as important, but it would be nice if it were still possible.
          Hide
          Erick Erickson added a comment -

          bq: so that solr.solr.home isn't cluttered

          I'm not sure about this either. There's no assumption when walking the tree that any instancedirs appear as immediate children of solr_home, you can have as many levels to the tree as you want. What is enforced is that when a core.properties file is found, Solr doesn't look any deeper. In other words there is no nesting of cores allowed. You can mix-and-match the level down the tree where the core is defined, subject to the above rule.

          So you could have a tree with only one directory in solr_home that was the root of a tree with all your cores like:

          <solr_home> cores
          level1a
          level1.1
          core1.1.1
          core1.1.2
          core1.1.3
          level1.2
          core2.1.1
          core2.1.2
          level2a
          core2a.1.1

          I'm not totally anti the idea, but would rather not start putting a bunch of stuff back into solr.xml if we can help it. Can this be made to work in your use-case or is there another reason besides clutter you'd want to root the cores someplace other than solr_home? Being able to cleanly separate the running solr from the data does come to mind, i.e. I might like to rm -rf <solr_home> and put in a new release without touching my data.....

          Show
          Erick Erickson added a comment - bq: so that solr.solr.home isn't cluttered I'm not sure about this either. There's no assumption when walking the tree that any instancedirs appear as immediate children of solr_home, you can have as many levels to the tree as you want. What is enforced is that when a core.properties file is found, Solr doesn't look any deeper. In other words there is no nesting of cores allowed. You can mix-and-match the level down the tree where the core is defined, subject to the above rule. So you could have a tree with only one directory in solr_home that was the root of a tree with all your cores like: <solr_home> cores level1a level1.1 core1.1.1 core1.1.2 core1.1.3 level1.2 core2.1.1 core2.1.2 level2a core2a.1.1 I'm not totally anti the idea, but would rather not start putting a bunch of stuff back into solr.xml if we can help it. Can this be made to work in your use-case or is there another reason besides clutter you'd want to root the cores someplace other than solr_home? Being able to cleanly separate the running solr from the data does come to mind, i.e. I might like to rm -rf <solr_home> and put in a new release without touching my data.....
          Hide
          Mark Miller added a comment -

          As far as xml format...I'm thinking along the lines of:

          note: persistent should be removed as an option.
          note: adminPath should no longer be configurable - solrcloud counts on it being that.
          note: defaultCoreName should probably also go away if using the new format - it's really a back compat thing that we don't want to support long term.

          <solr>
          
          
            <solrcloud>
              <str name="host">127.0.0.1</str>
              <str name="hostPort">${hostPort:8983}</str>
            </solrcloud>
          
            <shardHandlerFactory>
            ...
          </solr>
          
          Show
          Mark Miller added a comment - As far as xml format...I'm thinking along the lines of: note: persistent should be removed as an option. note: adminPath should no longer be configurable - solrcloud counts on it being that. note: defaultCoreName should probably also go away if using the new format - it's really a back compat thing that we don't want to support long term. <solr> <solrcloud> <str name="host">127.0.0.1</str> <str name="hostPort">${hostPort:8983}</str> </solrcloud> <shardHandlerFactory> ... </solr>
          Hide
          Shawn Heisey added a comment -

          There's no assumption when walking the tree that any instancedirs appear as immediate children

          That's good, and is awesome for detection of existing cores on startup, but what about features that automatically create new core directories? This already happens with the SolrCloud collection API, and if we implement the configs directory for config sets that aren't stored in zookeeper, it is likely to happen for non-Cloud deployments too.

          I am aware that my statements may seem somewhat disconnected, because in some ways they are. I'm dealing with two different Solr deployments that each have their own needs.

          1) I have a pretty small SolrCloud deployment with two servers plus a third zk node, numShards=1. I don't implement separation here, but it's not really needed. It currently places core directories right into solr.home, but I would like to change that.

          2) My main large Solr deployment doesn't use SolrCloud, and is not likely to use SolrCloud in the foreseeable future. That is where I have things heavily separated into cores, data, and config.

          In a cloud/zk deployment, you get dataDir separation for free, because the core config is elsewhere. This would also be the case with filesystem-based config sets. We therefore might not need to worry too much about the separation idea that I initially had, but I do think that it's important to have the cores go into a subdirectory by default.

          would rather not start putting a bunch of stuff back into solr.xml if we can help it

          +1 from me on that. Anything that we can move out of that file we should - basically make it so that only what's required at a global level to locate the rest of the resources Solr needs, plus any server-level config options that don't have an excellent all-purpose default. I think a 'coreDirectory' option falls into both of those categories. In a ZK-controlled world, a lot of global config options can be stored in ZK, but there are very good reasons to have a different coreDirectory option on each server.

          I might like to rm -rf <solr_home> and put in a new release without touching my data

          What I do for this is completely separate solr.home and CWD (jetty.home). CWD is /opt/solr4, solr.home is /index/solr4, and /index is a different filesystem. There are jars for jetty and log4j in /opt/solr4/lib, and extra jars for solr, lucene, and mysql in /index/solr4/lib.

          Show
          Shawn Heisey added a comment - There's no assumption when walking the tree that any instancedirs appear as immediate children That's good, and is awesome for detection of existing cores on startup, but what about features that automatically create new core directories? This already happens with the SolrCloud collection API, and if we implement the configs directory for config sets that aren't stored in zookeeper, it is likely to happen for non-Cloud deployments too. I am aware that my statements may seem somewhat disconnected, because in some ways they are. I'm dealing with two different Solr deployments that each have their own needs. 1) I have a pretty small SolrCloud deployment with two servers plus a third zk node, numShards=1. I don't implement separation here, but it's not really needed. It currently places core directories right into solr.home, but I would like to change that. 2) My main large Solr deployment doesn't use SolrCloud, and is not likely to use SolrCloud in the foreseeable future. That is where I have things heavily separated into cores, data, and config. In a cloud/zk deployment, you get dataDir separation for free, because the core config is elsewhere. This would also be the case with filesystem-based config sets. We therefore might not need to worry too much about the separation idea that I initially had, but I do think that it's important to have the cores go into a subdirectory by default. would rather not start putting a bunch of stuff back into solr.xml if we can help it +1 from me on that. Anything that we can move out of that file we should - basically make it so that only what's required at a global level to locate the rest of the resources Solr needs, plus any server-level config options that don't have an excellent all-purpose default. I think a 'coreDirectory' option falls into both of those categories. In a ZK-controlled world, a lot of global config options can be stored in ZK, but there are very good reasons to have a different coreDirectory option on each server. I might like to rm -rf <solr_home> and put in a new release without touching my data What I do for this is completely separate solr.home and CWD (jetty.home). CWD is /opt/solr4, solr.home is /index/solr4, and /index is a different filesystem. There are jars for jetty and log4j in /opt/solr4/lib, and extra jars for solr, lucene, and mysql in /index/solr4/lib.
          Hide
          Mark Miller added a comment -

          the solrcloud host is trouble spot - it will generally be different for each node - if this goes in zk, that type of config is still a problem - you have to use sys prop substitution and require system properties for setting these particular settings. I'm fine with that - I think you can still easily setup a prop file that gets auto sucked in for sys prop substitution as well.

          Show
          Mark Miller added a comment - the solrcloud host is trouble spot - it will generally be different for each node - if this goes in zk, that type of config is still a problem - you have to use sys prop substitution and require system properties for setting these particular settings. I'm fine with that - I think you can still easily setup a prop file that gets auto sucked in for sys prop substitution as well.
          Hide
          Erick Erickson added a comment -

          Does this structure work? I'll be working on this off and on this week, I've just about got a bunch of other stuff ready to check in and this is next up. Once this is done, I think we'll be in reasonably good shape, at least the major parts will be in place. Should we mark it "experimental" for 4.3?

          I figure to just look at the structure of the file to determine whether it's new or old style. It seems like the absence of a <cores> tag is enough to distinguish the two.

          And it would be really helpful if anyone spoke up right now about things I've forgotten, I culled things sitting in an airport with really poor internet connectivity.

          Of course any of the values can have system properties, I just didn't specify them here, and I've added coreRootDirectory as per Shawn's suggestion since the more I think about it the more it seems like a good thing to arbitrarily specify the root of where the directories live...

          <solr>
          <str name="hostContext">blah</str>
          <str name="sharedLib">lib</str>
          <int name="transientCacheSize">210</int>
          <str name="coreRootDirectory">absoluteoOrrelativePath</str>

          <solrcloud>
          <str name="host">127.0.0.1</str>
          <int name="hostPort">$

          {hostPort:8983}

          </int>
          <int name="zkClientTimeout">1500</int>
          <str name="hostContext">whatever</str>
          <str name=""></str>
          <str name=""></str>
          </solrcloud>

          <shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory">
          <int name="socketTimeout">$

          {socketTimeout:120000}

          </int>
          <int name="connTimeout">$

          {connTimeout:15000}

          </int>
          </shardHandlerFactory>
          </solr>

          Show
          Erick Erickson added a comment - Does this structure work? I'll be working on this off and on this week, I've just about got a bunch of other stuff ready to check in and this is next up. Once this is done, I think we'll be in reasonably good shape, at least the major parts will be in place. Should we mark it "experimental" for 4.3? I figure to just look at the structure of the file to determine whether it's new or old style. It seems like the absence of a <cores> tag is enough to distinguish the two. And it would be really helpful if anyone spoke up right now about things I've forgotten, I culled things sitting in an airport with really poor internet connectivity. Of course any of the values can have system properties, I just didn't specify them here, and I've added coreRootDirectory as per Shawn's suggestion since the more I think about it the more it seems like a good thing to arbitrarily specify the root of where the directories live... <solr> <str name="hostContext">blah</str> <str name="sharedLib">lib</str> <int name="transientCacheSize">210</int> <str name="coreRootDirectory">absoluteoOrrelativePath</str> <solrcloud> <str name="host">127.0.0.1</str> <int name="hostPort">$ {hostPort:8983} </int> <int name="zkClientTimeout">1500</int> <str name="hostContext">whatever</str> <str name=""></str> <str name=""></str> </solrcloud> <shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory"> <int name="socketTimeout">$ {socketTimeout:120000} </int> <int name="connTimeout">$ {connTimeout:15000} </int> </shardHandlerFactory> </solr>
          Hide
          Erick Erickson added a comment -

          I'm about to start working on this on the flight home, I hope to have this ready for review at least by the end of the weekend. I'll use the structure above unless there are howls of protest.
          Mark Miller and Shawn Heisey speak up!

          Show
          Erick Erickson added a comment - I'm about to start working on this on the flight home, I hope to have this ready for review at least by the end of the weekend. I'll use the structure above unless there are howls of protest. Mark Miller and Shawn Heisey speak up!
          Hide
          Mark Miller added a comment -

          Overall, +1, that seems good to me.

          coreRootDirectory

          With the idea of moving solr.xml to zk, coreRootDirectory seems like the prop we are talking about in SOLR-4697 - something similar to solr.home. I don't think it should block your work here, but I'd really hate to see that introduced in solr.xml. I think we need another mechanism for that level of prop - either the same one as solr.home, or one that takes advantage of solr.home (I gave one such example in my comment in SOLR-4697)

          Show
          Mark Miller added a comment - Overall, +1, that seems good to me. coreRootDirectory With the idea of moving solr.xml to zk, coreRootDirectory seems like the prop we are talking about in SOLR-4697 - something similar to solr.home. I don't think it should block your work here, but I'd really hate to see that introduced in solr.xml. I think we need another mechanism for that level of prop - either the same one as solr.home, or one that takes advantage of solr.home (I gave one such example in my comment in SOLR-4697 )
          Hide
          Mark Miller added a comment -

          I take that back on coreRootDirectory. It does seem like it belongs in solr.xml after some thought. Generally, you will use the same one on all of your nodes, and you can use system prop notation to override it per node if necessary. Sorry for the misdirection.

          Show
          Mark Miller added a comment - I take that back on coreRootDirectory. It does seem like it belongs in solr.xml after some thought. Generally, you will use the same one on all of your nodes, and you can use system prop notation to override it per node if necessary. Sorry for the misdirection.
          Hide
          Erick Erickson added a comment -

          NP, I had a boring plane ride home and got most of the structure in place. I have another plane ride tomorrow and bak on Friday, looks like I might have something ready to at least look at by Monday.

          Show
          Erick Erickson added a comment - NP, I had a boring plane ride home and got most of the structure in place. I have another plane ride tomorrow and bak on Friday, looks like I might have something ready to at least look at by Monday.
          Hide
          Erick Erickson added a comment -

          Preliminary patch with tests. I haven't looked it over thoroughly yet, and there's one failing test ZkCliTest. If Mark Miller has any clues what I managed to change here let me know, but I'm ready to stop for the night. I'll look at it tomorrow when it'll probably be obvious.

          It looks like I managed to have the default collection1 NOT be present or something, which leads to the discussion whether this should be assumed any more. Especially if we're deprecating defaultCoreName

          Show
          Erick Erickson added a comment - Preliminary patch with tests. I haven't looked it over thoroughly yet, and there's one failing test ZkCliTest. If Mark Miller has any clues what I managed to change here let me know, but I'm ready to stop for the night. I'll look at it tomorrow when it'll probably be obvious. It looks like I managed to have the default collection1 NOT be present or something, which leads to the discussion whether this should be assumed any more. Especially if we're deprecating defaultCoreName
          Hide
          Erick Erickson added a comment -

          Mark Miller Never mind. I'm not quite sure why I'm seeing what I'm seeing, but it's pretty clear that I screwed up the code. Told you it would be clear after sleeping.... Looking....

          Show
          Erick Erickson added a comment - Mark Miller Never mind. I'm not quite sure why I'm seeing what I'm seeing, but it's pretty clear that I screwed up the code. Told you it would be clear after sleeping.... Looking....
          Hide
          Erick Erickson added a comment -

          This may be ready. I'll let it sit for a day or two, go over it again, and commit it unless there are objections..

          All tests pass, I'll run nightly tests on it tonight.

          This patch and JIRA will NOT implement config sets, see SOLR-4478 for that. While I'd like to get 4478 into 4.3, it can wait until 4.4 if I don't have the time.

          Show
          Erick Erickson added a comment - This may be ready. I'll let it sit for a day or two, go over it again, and commit it unless there are objections.. All tests pass, I'll run nightly tests on it tonight. This patch and JIRA will NOT implement config sets, see SOLR-4478 for that. While I'd like to get 4478 into 4.3, it can wait until 4.4 if I don't have the time.
          Hide
          Erick Erickson added a comment -

          Put <solrcloud> section in solr.xml

          Show
          Erick Erickson added a comment - Put <solrcloud> section in solr.xml
          Hide
          Erick Erickson added a comment -

          NOTE: This final version of the patch is against 4x. I made a couple of trivial changes that didn't affect the code execution (he says), tested, and committed to trunk without making a final patch. Just so we have a record.

          Show
          Erick Erickson added a comment - NOTE: This final version of the patch is against 4x. I made a couple of trivial changes that didn't affect the code execution (he says), tested, and committed to trunk without making a final patch. Just so we have a record.
          Hide
          Erick Erickson added a comment -

          Trunk r: 1468284
          4x r: 1468289

          Let's open up new JIRAs for anything that needs to change.

          Show
          Erick Erickson added a comment - Trunk r: 1468284 4x r: 1468289 Let's open up new JIRAs for anything that needs to change.
          Hide
          Mark Miller added a comment -

          Finally getting to doing some review here.

          I'm -1 on the prevention of different cores having the same data directory. The lock factory is there to help there, and if you want to configure many cores to do read only against a datadir, you should be able to.

          Show
          Mark Miller added a comment - Finally getting to doing some review here. I'm -1 on the prevention of different cores having the same data directory. The lock factory is there to help there, and if you want to configure many cores to do read only against a datadir, you should be able to.
          Hide
          Mark Miller added a comment -

          Reopening - I'm going to take a close pass over this.

          Show
          Mark Miller added a comment - Reopening - I'm going to take a close pass over this.
          Hide
          Erick Erickson added a comment -

          Hmmm, OK, I can pull that out ASAP. How about a warning in the log file though? I think it's important to provide some way to debug this when it happens by accident rather than manual inspection of potentially a bazillion files.

          I take it there's no similar argument for core names? Having two cores with the same name seems...fraught.

          Show
          Erick Erickson added a comment - Hmmm, OK, I can pull that out ASAP. How about a warning in the log file though? I think it's important to provide some way to debug this when it happens by accident rather than manual inspection of potentially a bazillion files. I take it there's no similar argument for core names? Having two cores with the same name seems...fraught.
          Hide
          Mark Miller added a comment -

          I've got it, I'm working on a pass that has a few updates and fixes.

          How about a warning in the log file though?

          That seems fine to me.

          I take it there's no similar argument for core names? Having two cores with the same name seems...fraught.

          Right, that should fail.

          Show
          Mark Miller added a comment - I've got it, I'm working on a pass that has a few updates and fixes. How about a warning in the log file though? That seems fine to me. I take it there's no similar argument for core names? Having two cores with the same name seems...fraught. Right, that should fail.
          Hide
          Erick Erickson added a comment -

          Hmmm, I just pulled it out, thinking of putting this in 4725. On reflection, I also think it's a good idea to pull out the checking of same that I put in for old-style solr.xml, just log both conditions rather than change existing behavior.

          I'm about to put a patch up for SOLR-4725 (trunk), Do you want to just incorporate that or should I just drop it and let you handle it?

          Show
          Erick Erickson added a comment - Hmmm, I just pulled it out, thinking of putting this in 4725. On reflection, I also think it's a good idea to pull out the checking of same that I put in for old-style solr.xml, just log both conditions rather than change existing behavior. I'm about to put a patch up for SOLR-4725 (trunk), Do you want to just incorporate that or should I just drop it and let you handle it?
          Hide
          Mark Miller added a comment -

          I've got some work I'm in the middle of, and perhaps some more to come on further review. Most of it fairly little stuff, but also some critical stuff around using alternate root core dirs - that's what I'm involved with now, almost got it fixed and then I'll continue on and see what I find - I've already tackled a number of things though.

          Show
          Mark Miller added a comment - I've got some work I'm in the middle of, and perhaps some more to come on further review. Most of it fairly little stuff, but also some critical stuff around using alternate root core dirs - that's what I'm involved with now, almost got it fixed and then I'll continue on and see what I find - I've already tackled a number of things though.
          Hide
          Mark Miller added a comment -

          I'll quickly put up a check point patch and perhaps we can commit that to help keep in sync.

          Down the line I think we want to add some more tests for the alternate core root directory option.

          Show
          Mark Miller added a comment - I'll quickly put up a check point patch and perhaps we can commit that to help keep in sync. Down the line I think we want to add some more tests for the alternate core root directory option.
          Hide
          Erick Erickson added a comment -

          Well, if you're finding issues with the alternate core stuff, more tests are certainly in order <G>...

          I put up a patch on SOLR-4725 that takes out the failure case and logs a warning. For the legacy case it just logs warnings, but otherwise continues to act as before...

          The fiddling work there is just removing tests that should no longer fail....

          let me know when you're done and what you've incorporated and I'll reconcile, although today I won't be doning anything after 5:00 EST, I have some commitments.

          Show
          Erick Erickson added a comment - Well, if you're finding issues with the alternate core stuff, more tests are certainly in order <G>... I put up a patch on SOLR-4725 that takes out the failure case and logs a warning. For the legacy case it just logs warnings, but otherwise continues to act as before... The fiddling work there is just removing tests that should no longer fail.... let me know when you're done and what you've incorporated and I'll reconcile, although today I won't be doning anything after 5:00 EST, I have some commitments.
          Hide
          Mark Miller added a comment -

          Here is a patch from my first quick review of this - it fixes a few mostly minor issues. This is great work overall Erick!

          This fixes an issue with the core admin path on 5x, an issue with setting the right instance dir when using alternate core discovery locations, and some other minor nits.

          I plan to commit this very shortly so that Erick can merge his work in and then take a deeper dive through the evening.

          Show
          Mark Miller added a comment - Here is a patch from my first quick review of this - it fixes a few mostly minor issues. This is great work overall Erick! This fixes an issue with the core admin path on 5x, an issue with setting the right instance dir when using alternate core discovery locations, and some other minor nits. I plan to commit this very shortly so that Erick can merge his work in and then take a deeper dive through the evening.
          Hide
          Erick Erickson added a comment -

          OK, this has both change Mark and I were discussing. Tests pass, but I didn't think to make the change to the default persist before I ran the tests, so that change hasn't made it through all the tests and I have to leave....

          Show
          Erick Erickson added a comment - OK, this has both change Mark and I were discussing. Tests pass, but I didn't think to make the change to the default persist before I ran the tests, so that change hasn't made it through all the tests and I have to leave....
          Hide
          Erick Erickson added a comment -

          Mark checked the code in last night, merged into 4.x and 4.3.

          Show
          Erick Erickson added a comment - Mark checked the code in last night, merged into 4.x and 4.3.
          Hide
          Erick Erickson added a comment -

          commit tag bot seems to have skipped this...

          4.3 : r - 1469148
          4.x : r - 1469112
          trunk: r - 1469089

          Show
          Erick Erickson added a comment - commit tag bot seems to have skipped this... 4.3 : r - 1469148 4.x : r - 1469112 trunk: r - 1469089
          Hide
          Mark Miller added a comment -

          commit tag bot seems to have skipped this...

          It's not running - I've meant to move it from its ad hoc running spot (my primary dev machine where i have not been running it lately) to a permanent one, but just have not gotten to it yet. Soon though.

          Show
          Mark Miller added a comment - commit tag bot seems to have skipped this... It's not running - I've meant to move it from its ad hoc running spot (my primary dev machine where i have not been running it lately) to a permanent one, but just have not gotten to it yet. Soon though.
          Hide
          Jan Høydahl added a comment -

          Where did the sharedLib stuff go in the new solr.xml? Will it work with <str name="sharedLib">lib</str>? This should be documented in XML comments.

          Show
          Jan Høydahl added a comment - Where did the sharedLib stuff go in the new solr.xml? Will it work with <str name="sharedLib">lib</str> ? This should be documented in XML comments.
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Erick Erickson
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development