Solr
  1. Solr
  2. SOLR-5704

solr.xml coreNodeDirectory is ignored when creating new cores via REST(ish) apis

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 4.6.1
    • Fix Version/s: 4.7, 6.0
    • Component/s: None
    • Labels:
    • Environment:

      x86_64 Linux
      x86_64 Sun Java 7_u21

      Description

      "New style" core.properties auto-configuration works correctly at startup when ${coreRootDirectory} is specified in ${solr.home}/solr.xml, however it does not work if a core is later created dynamically via either (indirectly) the collection API or (directly) the core API.

      Core creation is always attempted in ${solr.home}.

      1. SOLR-5704.patch
        7 kB
        Alan Woodward
      2. SOLR-5704.patch
        3 kB
        Alan Woodward

        Activity

        Hide
        Jesse Sipprell added a comment - - edited

        The following is the common solr.xml we use on all 4.6.1 nodes with path and hostname redactions for security reasons:

        <solr>
          <str name="adminHandler">${adminHandler:org.apache.solr.handler.admin.CoreAdminHandler}</str>
          <int name="coreLoadThreads">${coreLoadThreads:3}</int>
          <str name="coreRootDirectory">${coreRootDirectory:/data/solr/cores/}</str> <!-- usually solrHome  -->
          <str name="managementPath">${managementPath:admin}</str>
          <str name="sharedLib">${sharedLib:lib}</str>
          <str name="shareSchema">${shareSchema:false}</str>
          <!-- transientCacheSize is ignored for everything but transient cores. -->
          <int name="transientCacheSize">${transientCacheSize:2147483647}</int>
          <solrcloud>
            <int name="distribUpdateConnTimeout">${distribUpdTimeout:30000}</int>
            <int name="distribUpdateSoTimeout">${distribUpdateTimeout:15000}</int>
            <int name="leaderVoteWait">${leaderVoteWait:30000}</int>
            <str name="host">${host:a-fully-qualified-hostname.mcclatchyinteractive.com}</str>
            <str name="hostContext">${hostContext:solr}</str>
            <int name="hostPort">${jetty.port:8983}</int>
            <int name="zkClientTimeout">${zkClientTimeout:15000}</int>
            <str name="zkHost">${zkHost:a,zk,cluster,host,port,namespace,list}</str>
            <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
          </solrcloud>
        </solr>
        
        Show
        Jesse Sipprell added a comment - - edited The following is the common solr.xml we use on all 4.6.1 nodes with path and hostname redactions for security reasons: <solr> <str name= "adminHandler" > ${adminHandler:org.apache.solr.handler.admin.CoreAdminHandler} </str> <int name= "coreLoadThreads" > ${coreLoadThreads:3} </int> <str name= "coreRootDirectory" > ${coreRootDirectory:/data/solr/cores/} </str> <!-- usually solrHome --> <str name= "managementPath" > ${managementPath:admin} </str> <str name= "sharedLib" > ${sharedLib:lib} </str> <str name= "shareSchema" > ${shareSchema:false} </str> <!-- transientCacheSize is ignored for everything but transient cores. --> <int name= "transientCacheSize" > ${transientCacheSize:2147483647} </int> <solrcloud> <int name= "distribUpdateConnTimeout" > ${distribUpdTimeout:30000} </int> <int name= "distribUpdateSoTimeout" > ${distribUpdateTimeout:15000} </int> <int name= "leaderVoteWait" > ${leaderVoteWait:30000} </int> <str name= "host" > ${host:a-fully-qualified-hostname.mcclatchyinteractive.com} </str> <str name= "hostContext" > ${hostContext:solr} </str> <int name= "hostPort" > ${jetty.port:8983} </int> <int name= "zkClientTimeout" > ${zkClientTimeout:15000} </int> <str name= "zkHost" > ${zkHost:a,zk,cluster,host,port,namespace,list} </str> <bool name= "genericCoreNodeNames" > ${genericCoreNodeNames:true} </bool> </solrcloud> </solr>
        Hide
        Erick Erickson added a comment -

        Alan is picking this up, but a tangential thing....

        The transientCacheSize is completely out of what we had in mind when
        the code was put together. You'll effectively never age out any transient
        cores. Is that intentional?

        Show
        Erick Erickson added a comment - Alan is picking this up, but a tangential thing.... The transientCacheSize is completely out of what we had in mind when the code was put together. You'll effectively never age out any transient cores. Is that intentional?
        Hide
        Jesse Sipprell added a comment -

        The solr.xml I posted was initially just copy-and-pasted from some site that documented it (which also suggested that these values not be altered). The only values we have changed are coreRootDirectory, host, zkHost and (I think) zkClientTimeout.

        We don't use transient cores so the value isn't terribly relevant to us. In fact, we use configuration management systems to manage core directories and core.properties files in a totally different location separate from solr.home and solr.xml. The only reason this (the problem discussed in this ticket) is an issue for us is because it makes it impossible to perform dynamic shardsplitting as the user we run Solr as doesn't have write access to solr.home.

        Thanks,

        Jesse

        Show
        Jesse Sipprell added a comment - The solr.xml I posted was initially just copy-and-pasted from some site that documented it (which also suggested that these values not be altered). The only values we have changed are coreRootDirectory , host , zkHost and (I think) zkClientTimeout . We don't use transient cores so the value isn't terribly relevant to us. In fact, we use configuration management systems to manage core directories and core.properties files in a totally different location separate from solr.home and solr.xml . The only reason this (the problem discussed in this ticket) is an issue for us is because it makes it impossible to perform dynamic shardsplitting as the user we run Solr as doesn't have write access to solr.home . Thanks, Jesse
        Hide
        Alan Woodward added a comment -

        So the problem here is that when a CoreDescriptor gets a relative instance dir, it always resolves that against solr home. This patch updates it to check for a coreRootDirectory first.

        Show
        Alan Woodward added a comment - So the problem here is that when a CoreDescriptor gets a relative instance dir, it always resolves that against solr home. This patch updates it to check for a coreRootDirectory first.
        Hide
        Jesse Sipprell added a comment -

        I've tested out the attached patch and it seems to work as expected; at least with SPLITSHARD.

        Thanks,

        Jesse

        Show
        Jesse Sipprell added a comment - I've tested out the attached patch and it seems to work as expected; at least with SPLITSHARD. Thanks, Jesse
        Hide
        Alan Woodward added a comment -

        Updated patch, fixing a couple of test fails due to TestHarness creating a ConfigSolr object with a null resource loader.

        Show
        Alan Woodward added a comment - Updated patch, fixing a couple of test fails due to TestHarness creating a ConfigSolr object with a null resource loader.
        Hide
        ASF subversion and git services added a comment -

        Commit 1566598 from Alan Woodward in branch 'dev/trunk'
        [ https://svn.apache.org/r1566598 ]

        SOLR-5704: new cores should be created under coreRootDirectory

        Show
        ASF subversion and git services added a comment - Commit 1566598 from Alan Woodward in branch 'dev/trunk' [ https://svn.apache.org/r1566598 ] SOLR-5704 : new cores should be created under coreRootDirectory
        Hide
        ASF subversion and git services added a comment -

        Commit 1566600 from Alan Woodward in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1566600 ]

        SOLR-5704: new cores should be created under coreRootDirectory

        Show
        ASF subversion and git services added a comment - Commit 1566600 from Alan Woodward in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1566600 ] SOLR-5704 : new cores should be created under coreRootDirectory
        Hide
        Alan Woodward added a comment -

        Thanks Jesse!

        Show
        Alan Woodward added a comment - Thanks Jesse!
        Hide
        Erick Erickson added a comment -

        Thanks guys!

        Show
        Erick Erickson added a comment - Thanks guys!
        Hide
        Shawn Heisey added a comment -

        A user on IRC is running 4.9.0 and claims that new cores are created in the solr home rather than the configured coreRootDirectory, so on restart they are not being discovered. I'm doing what I can to be sure there are no jars from a previous version hanging around.

        Show
        Shawn Heisey added a comment - A user on IRC is running 4.9.0 and claims that new cores are created in the solr home rather than the configured coreRootDirectory, so on restart they are not being discovered. I'm doing what I can to be sure there are no jars from a previous version hanging around.
        Hide
        Shawn Heisey added a comment -

        Looks like a new issue. I'll get that filed.

        11:53 < _sp0k_> It indeed is reading coreRootDirectory, but due to missing
                        trailer collection directory is created wrong. e.g.
                        coreRootDirectory=/usr/local/solr/cores, collection directory
                        created by API would be -
                        /usr/local/solr/cores${collection}_shard_replica where it
                        should be /usr/local/solr/cores/${collection}_shard_replica
        
        Show
        Shawn Heisey added a comment - Looks like a new issue. I'll get that filed. 11:53 < _sp0k_> It indeed is reading coreRootDirectory, but due to missing trailer collection directory is created wrong. e.g. coreRootDirectory=/usr/local/solr/cores, collection directory created by API would be - /usr/local/solr/cores${collection}_shard_replica where it should be /usr/local/solr/cores/${collection}_shard_replica

          People

          • Assignee:
            Alan Woodward
            Reporter:
            Jesse Sipprell
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development