Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 4.3, 5.0
    • Component/s: multicore
    • Labels:
      None

      Description

      If there are 1000's of cores the no:of times we need to load and parse the solrconfig.xml is going to be very expensive. It is desirable to just create one instance of SolrConfig object and re-use it across cores

      1. SOLR-919.patch
        15 kB
        Noble Paul

        Issue Links

          Activity

          Noble Paul created issue -
          Noble Paul made changes -
          Field Original Value New Value
          Link This issue blocks SOLR-920 [ SOLR-920 ]
          Noble Paul made changes -
          Summary Cache and reuse the DOM object of solrconfig.xml Cache and reuse the SolrConfig
          Assignee Noble Paul [ noble.paul ]
          Fix Version/s 1.5 [ 12313566 ]
          Description If there are 1000's of cores the no:of times we need to load and parse the solrconfig.xml is going to be very expensive. In such a case we may mostly have same solrconfig.xml with some core properties embedded

          The solution is , if the location of the xml file is same (on disk) let us keep only one DOM object and reuse it for every other cores. So we can keep a cache at the CoreContainer level .

          We save file reading , XML parsing and RAM in one go

          The only challenge here is that it is we currently substitute core properties at read time. We will have to postpone this to consumption time. So whenever we read the values from the DOM the values can be substituted and used

          If there are 1000's of cores the no:of times we need to load and parse the solrconfig.xml is going to be very expensive. It is desirable to just create one instance of SolrConfig object and re-use it across cores


          Noble Paul made changes -
          Link This issue depends on SOLR-1198 [ SOLR-1198 ]
          Shalin Shekhar Mangar made changes -
          Link This issue blocks SOLR-1293 [ SOLR-1293 ]
          Shalin Shekhar Mangar made changes -
          Component/s multicore [ 12313102 ]
          Hide
          Noble Paul added a comment -

          untested patch

          Show
          Noble Paul added a comment - untested patch
          Noble Paul made changes -
          Attachment SOLR-919.patch [ 12428035 ]
          Hide
          Hoss Man added a comment -

          Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

          http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

          Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

          A unique token for finding these 240 issues in the future: hossversioncleanup20100527

          Show
          Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
          Hoss Man made changes -
          Fix Version/s Next [ 12315093 ]
          Fix Version/s 1.5 [ 12313566 ]
          Hoss Man made changes -
          Fix Version/s 3.2 [ 12316172 ]
          Fix Version/s Next [ 12315093 ]
          Hide
          Robert Muir added a comment -

          Bulk move 3.2 -> 3.3

          Show
          Robert Muir added a comment - Bulk move 3.2 -> 3.3
          Robert Muir made changes -
          Fix Version/s 3.3 [ 12316471 ]
          Fix Version/s 3.2 [ 12316172 ]
          Robert Muir made changes -
          Fix Version/s 3.4 [ 12316683 ]
          Fix Version/s 4.0 [ 12314992 ]
          Fix Version/s 3.3 [ 12316471 ]
          Hide
          Robert Muir added a comment -

          3.4 -> 3.5

          Show
          Robert Muir added a comment - 3.4 -> 3.5
          Robert Muir made changes -
          Fix Version/s 3.5 [ 12317876 ]
          Fix Version/s 3.4 [ 12316683 ]
          Simon Willnauer made changes -
          Fix Version/s 3.6 [ 12319065 ]
          Fix Version/s 3.5 [ 12317876 ]
          Hide
          Drew Morris added a comment -

          Shouldn't this work the same way as SOLR-920? It would seem the logic applies to both bugs.

          Show
          Drew Morris added a comment - Shouldn't this work the same way as SOLR-920 ? It would seem the logic applies to both bugs.
          Hide
          Noble Paul added a comment -

          @Drew , The ;ogic is mostly same, but there are some more changes required in the core to make it work because a few references are kept in SolrConfig which should have been keptin the core

          Show
          Noble Paul added a comment - @Drew , The ;ogic is mostly same, but there are some more changes required in the core to make it work because a few references are kept in SolrConfig which should have been keptin the core
          Robert Muir made changes -
          Fix Version/s 3.6 [ 12319065 ]
          Hide
          Hoss Man added a comment -

          bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment

          Show
          Hoss Man added a comment - bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment
          Hoss Man made changes -
          Fix Version/s 4.0 [ 12322455 ]
          Fix Version/s 4.0-ALPHA [ 12314992 ]
          Hide
          Robert Muir added a comment -

          rmuir20120906-bulk-40-change

          Show
          Robert Muir added a comment - rmuir20120906-bulk-40-change
          Robert Muir made changes -
          Fix Version/s 4.0 [ 12322551 ]
          Fix Version/s 4.0-BETA [ 12322455 ]
          Hide
          Robert Muir added a comment -

          moving all 4.0 issues not touched in a month to 4.1

          Show
          Robert Muir added a comment - moving all 4.0 issues not touched in a month to 4.1
          Robert Muir made changes -
          Fix Version/s 4.1 [ 12321141 ]
          Fix Version/s 4.0 [ 12322551 ]
          Hide
          Erick Erickson added a comment -

          Taking some very rough timings on a 2012 Macbook pro, loading the config file takes something on the order of 300-400 ms, so for 1,000s of cores we need this.

          Show
          Erick Erickson added a comment - Taking some very rough timings on a 2012 Macbook pro, loading the config file takes something on the order of 300-400 ms, so for 1,000s of cores we need this.
          Mark Miller made changes -
          Fix Version/s 4.2 [ 12323893 ]
          Fix Version/s 5.0 [ 12321664 ]
          Fix Version/s 4.1 [ 12321141 ]
          Erick Erickson made changes -
          Assignee Noble Paul [ noble.paul ] Erick Erickson [ erickerickson ]
          Hide
          Erick Erickson added a comment -

          this will be part of SOLR-4478 (named config sets)

          Show
          Erick Erickson added a comment - this will be part of SOLR-4478 (named config sets)
          Erick Erickson made changes -
          Link This issue is superceded by SOLR-4478 [ SOLR-4478 ]
          Erick Erickson made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 4.3 [ 12324128 ]
          Fix Version/s 4.2 [ 12323893 ]
          Resolution Won't Fix [ 2 ]
          Gavin made changes -
          Link This issue blocks SOLR-920 [ SOLR-920 ]
          Gavin made changes -
          Link This issue is depended upon by SOLR-920 [ SOLR-920 ]
          Gavin made changes -
          Link This issue depends on SOLR-1198 [ SOLR-1198 ]
          Gavin made changes -
          Link This issue depends upon SOLR-1198 [ SOLR-1198 ]
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.
          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Erick Erickson
              Reporter:
              Noble Paul
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development