Solr
  1. Solr
  2. SOLR-880

SolrCore should have a a lazy startup option

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: multicore
    • Labels:
      None

      Description

      • a core should have an option of loadOnStartup=true|false. default should be true

      If there are too many cores (tens of thousands) where each of them may be used occassionally, we should not load all of them at once. In the runtime I should be able to STOP and START a core on demand. A listing command would let me know which one is present and what is up and what is down. A stopped core must not use any resource

      1. SOLR-880.patch
        38 kB
        Erick Erickson
      2. SOLR-880.patch
        79 kB
        Erick Erickson

        Issue Links

          Activity

          Hide
          Mike Anderson added a comment -

          I'm not entirely clear on how START would differ from CREATE, and how STOP would differ from UNLOAD. I gather there are certain tasks that occur in CREATE that would be skipped in a START command, and that a START command could only be issued on a STOPPED core, which had previously been CREATED (but not UNLOADED).

          This issue hasn't been touched in over a year, is it still thought to be the right approach to improving multicore? What are the saving of START/STOP vs CREATE/UNLOAD? is it an issue of speed? memory?

          Show
          Mike Anderson added a comment - I'm not entirely clear on how START would differ from CREATE, and how STOP would differ from UNLOAD. I gather there are certain tasks that occur in CREATE that would be skipped in a START command, and that a START command could only be issued on a STOPPED core, which had previously been CREATED (but not UNLOADED). This issue hasn't been touched in over a year, is it still thought to be the right approach to improving multicore? What are the saving of START/STOP vs CREATE/UNLOAD? is it an issue of speed? memory?
          Hide
          Erick Erickson added a comment -

          First cut at the basics, putting up a preliminary version for comments.

          The general approach here is that, for any lazy cores, keep a separate list of SolrCoreDescriptors. When we get a core, if it's not already loaded, look in this separate list and create it at that point.

          Note a bunch of things:

          1> many of the changes in CoreContainer are that I factored out creating cores from local files and Zookeeper into two methods, I was having a hard time keeping the zk and non-zk bits separate.

          2> There are some TODOs and EOEs that I have to take out.

          3> I'm not all that happy with the tests, especially making new config directories just for this case with tests. But I was going a bit crazy yesterday trying to use the "usual" methods for writing tests, but as far as I can tell, there are built-in assumptions in things like TestHarness that don't work well with different cores. Any suggestions?

          4> All test pass. I fired up an example in our standard multicore system, and it's actually kinda cool. The admin console doesn't show the lazy core, but I can index to it with post.jar, then the admin screen shows it and I can query it. I can shut down and restart and the first query on the lazy core then returns results, even though it again isn't in the admin screen.

          5> I haven't tested this all that thoroughly, this is preliminary for comments. This is part of SOLR-1293.

          6> Next up is SOLR-1028, limiting the number of cores that can be loaded simultaneously.

          7> I'm quite sure I'll screw up the reference counting and/or there are nooks and crannies that I don't even know exist. Please let me know of any off the tops of your heads!

          8> All tests pass. Can I ship it now? <G>

          Show
          Erick Erickson added a comment - First cut at the basics, putting up a preliminary version for comments. The general approach here is that, for any lazy cores, keep a separate list of SolrCoreDescriptors. When we get a core, if it's not already loaded, look in this separate list and create it at that point. Note a bunch of things: 1> many of the changes in CoreContainer are that I factored out creating cores from local files and Zookeeper into two methods, I was having a hard time keeping the zk and non-zk bits separate. 2> There are some TODOs and EOEs that I have to take out. 3> I'm not all that happy with the tests, especially making new config directories just for this case with tests. But I was going a bit crazy yesterday trying to use the "usual" methods for writing tests, but as far as I can tell, there are built-in assumptions in things like TestHarness that don't work well with different cores. Any suggestions? 4> All test pass. I fired up an example in our standard multicore system, and it's actually kinda cool. The admin console doesn't show the lazy core, but I can index to it with post.jar, then the admin screen shows it and I can query it. I can shut down and restart and the first query on the lazy core then returns results, even though it again isn't in the admin screen. 5> I haven't tested this all that thoroughly, this is preliminary for comments. This is part of SOLR-1293 . 6> Next up is SOLR-1028 , limiting the number of cores that can be loaded simultaneously. 7> I'm quite sure I'll screw up the reference counting and/or there are nooks and crannies that I don't even know exist. Please let me know of any off the tops of your heads! 8> All tests pass. Can I ship it now? <G>
          Hide
          Erick Erickson added a comment -

          Removed STOP from description, functionality is handled by UNLOAD

          Broke out the "add a list command" to it's own JIRA, see: https://issues.apache.org/jira/browse/SOLR-3980

          Show
          Erick Erickson added a comment - Removed STOP from description, functionality is handled by UNLOAD Broke out the "add a list command" to it's own JIRA, see: https://issues.apache.org/jira/browse/SOLR-3980
          Hide
          Erick Erickson added a comment -

          New version. Removed TODOs and my initials.

          > Got rid of the extra test directory that I wasn't happy with anyway.

          > Took a whack at returning SolrExceptions from CoreContainer. This required that I change a number of tests, I'd particularly appreciate anyone looking at that whole thing.

          > All tests pass, I'll commit this in a couple of days if nobody objects.

          Show
          Erick Erickson added a comment - New version. Removed TODOs and my initials. > Got rid of the extra test directory that I wasn't happy with anyway. > Took a whack at returning SolrExceptions from CoreContainer. This required that I change a number of tests, I'd particularly appreciate anyone looking at that whole thing. > All tests pass, I'll commit this in a couple of days if nobody objects.
          Hide
          Erick Erickson added a comment -

          This functionality will be part of SOLR-1028, which will be checked in shortly.

          Show
          Erick Erickson added a comment - This functionality will be part of SOLR-1028 , which will be checked in shortly.

            People

            • Assignee:
              Erick Erickson
              Reporter:
              Noble Paul
            • Votes:
              8 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development