Solr
  1. Solr
  2. SOLR-1575

loading solr.xml takes a lot of time when there are lot of cores

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: multicore
    • Labels:
      None

      Description

      We have around 38K cores which takes up around 25 mins to start the server. Keep in mind that no cores are started and this is the time taken for loading the solr.xml.

        Issue Links

          Activity

          Hide
          Noble Paul added a comment -

          It may not be due to solr.xml itself. Further investigation may reveal what is wrong..

          Show
          Noble Paul added a comment - It may not be due to solr.xml itself. Further investigation may reveal what is wrong..
          Hide
          olivier soyez added a comment -

          We have around 50K cores with loadOnStart="false" and same issue to start the server.
          It takes a very long time to start, and it is just the time taken for loading the config file listing the available solr cores : solr.xml.

          To solve this, in the "load" function, I just replace the line 532 of CoreContainer.java (Apache Solr 4.0 release) from "p.setCoreProperties(readProperties(cfg, node));" to "p.setCoreProperties(readProperties(cfg, node.getParentNode().removeChild(node)));"

          Now, it takes a few seconds to start the server

          Show
          olivier soyez added a comment - We have around 50K cores with loadOnStart="false" and same issue to start the server. It takes a very long time to start, and it is just the time taken for loading the config file listing the available solr cores : solr.xml. To solve this, in the "load" function, I just replace the line 532 of CoreContainer.java (Apache Solr 4.0 release) from "p.setCoreProperties(readProperties(cfg, node));" to "p.setCoreProperties(readProperties(cfg, node.getParentNode().removeChild(node)));" Now, it takes a few seconds to start the server
          Hide
          Erick Erickson added a comment -

          This is very much changing, but be aware that the "loadOnStartup" is only for trunk (5.0). It hasn't been back-ported to the 4.x code line yet, so it's unsurprising that it doesn't make any difference.

          It would be great if you could pull down 5x and give it a whirl without your change as a test... but be sure you have loadOnStartup rather than loadOnStart.

          Take a look at SOLR-4083 for the direction things are going.

          Show
          Erick Erickson added a comment - This is very much changing, but be aware that the "loadOnStartup" is only for trunk (5.0). It hasn't been back-ported to the 4.x code line yet, so it's unsurprising that it doesn't make any difference. It would be great if you could pull down 5x and give it a whirl without your change as a test... but be sure you have loadOnStartup rather than loadOnStart. Take a look at SOLR-4083 for the direction things are going.
          Hide
          olivier soyez added a comment -

          In fact, I used the 4.0 code + SOLR-1293 modified code to test lotsofcores solr 4.0 (to be able to handle 50k cores per server with 1000 lives cores max at any time, shared configs, numBuckets, lazy load cores, efficient start cores, automatic start/stop cores, core creation on the fly, efficient fresh start). It's the reason why I have the loadOnStart="false" option.

          Anyways, I pull down 5x and without changes, I try to start the server (50K cores with "loadOnStartup=false"), and same issue (very very long time to start ...)
          After that, I applied the change (line 590 of CoreContainer.java Apache Solr 5.0 trunck). In this way, the node you are working with is detached from the main document. So, it's a huge speed up for the evaluate method used in readProperties function called for each core, especially in case of lots of cores. And now it takes a few seconds to start the server.

          Thanks, I would following SOLR-4083, because in your specific use case (50k cores), deprecate solr.xml entirely could be a scalability problem (especially fresh start time).

          Show
          olivier soyez added a comment - In fact, I used the 4.0 code + SOLR-1293 modified code to test lotsofcores solr 4.0 (to be able to handle 50k cores per server with 1000 lives cores max at any time, shared configs, numBuckets, lazy load cores, efficient start cores, automatic start/stop cores, core creation on the fly, efficient fresh start). It's the reason why I have the loadOnStart="false" option. Anyways, I pull down 5x and without changes, I try to start the server (50K cores with "loadOnStartup=false"), and same issue (very very long time to start ...) After that, I applied the change (line 590 of CoreContainer.java Apache Solr 5.0 trunck). In this way, the node you are working with is detached from the main document. So, it's a huge speed up for the evaluate method used in readProperties function called for each core, especially in case of lots of cores. And now it takes a few seconds to start the server. Thanks, I would following SOLR-4083 , because in your specific use case (50k cores), deprecate solr.xml entirely could be a scalability problem (especially fresh start time).
          Hide
          Erick Erickson added a comment -

          Well, first the option is "loadOnStartup", not "loadOnStart". loadOnStart="false" will still load everything. If you change that and still get extremely long load times, let me know....

          Second, I've tested 15K cores parsing on a spinning-disk and it "seemed" acceptable to me performance wise, 15-20 seconds to get all the files off the disk (but not load cores).

          But real-life tests always trump experiments, please let me know if it continues to not work....

          Show
          Erick Erickson added a comment - Well, first the option is "loadOnStartup", not "loadOnStart". loadOnStart="false" will still load everything. If you change that and still get extremely long load times, let me know.... Second, I've tested 15K cores parsing on a spinning-disk and it "seemed" acceptable to me performance wise, 15-20 seconds to get all the files off the disk (but not load cores). But real-life tests always trump experiments, please let me know if it continues to not work....
          Hide
          olivier soyez added a comment -

          With trunk solr 5.0, all cores with loadOnStartup="false" in solr.xml :

          • without any changes in the code :
            15K cores server startup in 220115 ms ~ 4 min
            50K cores server startup in 2925056 ms ~ 49 min
          • with the change :
            15K cores server startup in 8145 ms ~ 8 sec
            50K cores server startup in 17211 ms ~ 17 sec

          This little change is a big boost for the server startup ...

          Show
          olivier soyez added a comment - With trunk solr 5.0, all cores with loadOnStartup="false" in solr.xml : without any changes in the code : 15K cores server startup in 220115 ms ~ 4 min 50K cores server startup in 2925056 ms ~ 49 min with the change : 15K cores server startup in 8145 ms ~ 8 sec 50K cores server startup in 17211 ms ~ 17 sec This little change is a big boost for the server startup ...

            People

            • Assignee:
              Unassigned
              Reporter:
              Noble Paul
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development