Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      if there are 1000's of cores then the cost of loading unloading schema.xml can be prohibitive
      similar to SOLR-919 we can also cache the DOM object of schema.xml if the location on disk is same. All the dynamic properties can be replaced lazily when they are read.

      We can go one step ahead in this case. Th IndexSchema object is immutable . So if there are no core properties then the same IndexSchema object can be used across all the cores

      1. SOLR-920.patch
        1 kB
        Noble Paul
      2. SOLR-920.patch
        1 kB
        Noble Paul
      3. SOLR-920.patch
        2 kB
        Noble Paul

        Issue Links

          Activity

          Hide
          Noble Paul added a comment -

          how about an extra attribute in the cores tag to enable sharing as follows.

          <solr>
            <cores adminPath="/admin/cores" shareSchema="true">
            </cores>
          </solr>
          

          if shareSchema is set to true then for a given path only one instance of IndexSchema is created and it will be shared . It is the responsibilty of the user to ensure that he does not use any core specific properties in the schema.xml when this attribute is set to true

          Show
          Noble Paul added a comment - how about an extra attribute in the cores tag to enable sharing as follows. <solr> <cores adminPath= "/admin/cores" shareSchema= "true" > </cores> </solr> if shareSchema is set to true then for a given path only one instance of IndexSchema is created and it will be shared . It is the responsibilty of the user to ensure that he does not use any core specific properties in the schema.xml when this attribute is set to true
          Hide
          Otis Gospodnetic added a comment -

          Looks good to me. What happens when a core has a copy of schema.xml in its conf/ dir and that schema.xml is potentially different from the shared one?

          Show
          Otis Gospodnetic added a comment - Looks good to me. What happens when a core has a copy of schema.xml in its conf/ dir and that schema.xml is potentially different from the shared one?
          Hide
          Noble Paul added a comment -

          What happens when a core has a copy of schema.xml in its conf/ dir and that schema.xml is potentially different from the shared one?

          for a given path on disk one instance of IndexSchema is maintained in memory. The cache looks as follows .

          // in CoreContainer.java. 
          //The key is the absolute path to the schema.xml. The value is the IndexSchema instance.
          private Map<String,IndexSchema> schemaCache;
          

          It is also possible to maintain a timestamp of the file also in the key (say /data/solr/confschema.xml:yyyyMMddhhmmss)so that if the file is modified a new instance of IndexScema can be created

          Show
          Noble Paul added a comment - What happens when a core has a copy of schema.xml in its conf/ dir and that schema.xml is potentially different from the shared one? for a given path on disk one instance of IndexSchema is maintained in memory. The cache looks as follows . // in CoreContainer.java. //The key is the absolute path to the schema.xml. The value is the IndexSchema instance. private Map< String ,IndexSchema> schemaCache; It is also possible to maintain a timestamp of the file also in the key (say /data/solr/confschema.xml:yyyyMMddhhmmss)so that if the file is modified a new instance of IndexScema can be created
          Hide
          Otis Gospodnetic added a comment -

          So if my core has its own schema.xml in the right place (in conf/schema.xml), that schema will be used, not the shard one?

          Show
          Otis Gospodnetic added a comment - So if my core has its own schema.xml in the right place (in conf/schema.xml), that schema will be used, not the shard one?
          Hide
          Noble Paul added a comment -

          So if my core has its own schema.xml in the right place (in conf/schema.xml), that schema will be used, not the shard one?

          right. one instance per (file+version).

          Show
          Noble Paul added a comment - So if my core has its own schema.xml in the right place (in conf/schema.xml), that schema will be used, not the shard one? right. one instance per (file+version).
          Hide
          Noble Paul added a comment -

          I plan to commit this in a day or two

          Show
          Noble Paul added a comment - I plan to commit this in a day or two
          Hide
          Noble Paul added a comment -

          committed revision:778652

          Show
          Noble Paul added a comment - committed revision:778652
          Hide
          Noble Paul added a comment -

          the attribute is not persisted back to solr.xml

          Show
          Noble Paul added a comment - the attribute is not persisted back to solr.xml
          Hide
          Noble Paul added a comment -

          logging that re-use is done and the path is made right

          Show
          Noble Paul added a comment - logging that re-use is done and the path is made right
          Hide
          Noble Paul added a comment -

          committed :r783631

          Show
          Noble Paul added a comment - committed :r783631
          Hide
          Shalin Shekhar Mangar added a comment -

          I think this change is not mentioned in the CHANGES.txt? Also, the fix version should be marked as 1.4.

          Show
          Shalin Shekhar Mangar added a comment - I think this change is not mentioned in the CHANGES.txt? Also, the fix version should be marked as 1.4.
          Hide
          Yonik Seeley added a comment -

          Just got around to reviewing this.... seems like it's an unbounded cache?
          If someone kept modifying the schema and reloading the core (as can happen with replication too), then all the older versions hang around in the cache forever, right?

          Show
          Yonik Seeley added a comment - Just got around to reviewing this.... seems like it's an unbounded cache? If someone kept modifying the schema and reloading the core (as can happen with replication too), then all the older versions hang around in the cache forever, right?
          Hide
          Noble Paul added a comment -

          seems like it's an unbounded cache?

          yes. It is unbounded. Let us make it bounded

          Show
          Noble Paul added a comment - seems like it's an unbounded cache? yes. It is unbounded. Let us make it bounded
          Hide
          Drew Morris added a comment -

          Just reviewing the syntax of the XML... if you have 2 different schemas that should be shared with multiple cores... will this still work? So if I had 1000 cores sharing 1 schema and 1000 cores sharing a different schema will that be possible with this implementation?

          Show
          Drew Morris added a comment - Just reviewing the syntax of the XML... if you have 2 different schemas that should be shared with multiple cores... will this still work? So if I had 1000 cores sharing 1 schema and 1000 cores sharing a different schema will that be possible with this implementation?
          Hide
          Noble Paul added a comment -

          yes . it is possible

          Show
          Noble Paul added a comment - yes . it is possible
          Hide
          Noble Paul added a comment -

          , then all the older versions hang around in the cache forever, right?

          nope. The cache key is the FQN of the file . in replication , the same file is overwritten. So the cache entry is overwritten

          Show
          Noble Paul added a comment - , then all the older versions hang around in the cache forever, right? nope. The cache key is the FQN of the file . in replication , the same file is overwritten. So the cache entry is overwritten

            People

            • Assignee:
              Noble Paul
              Reporter:
              Noble Paul
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development