Solr
  1. Solr
  2. SOLR-1304

Make it possible to force replication of at least some of the config files even if the index hasn't changed

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 4.9, 5.0
    • Component/s: replication (java)
    • Labels:
      None

      Description

      From http://markmail.org/thread/vpk2fsjns7u2uopd

      Here is a use case:

      • Index is mostly static (nightly updates)
      • elevate.xml needs to be changed throughout the day
      • elevate.xml needs to be pushed to slaves and solr needs to reload it

      This is currently not possible because replication will happen only if the index
      changed in some way. You can't force a commit to fake index change. So one has
      to either:

      • add/delete dummy docs on master to force index change
      • write an external script that copies the config file to slaves

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment -

          Move issue to Solr 4.9.

          Show
          Uwe Schindler added a comment - Move issue to Solr 4.9.
          Hide
          Steve Rowe added a comment -

          Bulk move 4.4 issues to 4.5 and 5.0

          Show
          Steve Rowe added a comment - Bulk move 4.4 issues to 4.5 and 5.0
          Hide
          Fredrik Rodland added a comment -

          +1 - really need this.

          Show
          Fredrik Rodland added a comment - +1 - really need this.
          Hide
          Hoss Man added a comment -

          Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently.

          email notification suppressed to prevent mass-spam
          psuedo-unique token identifying these issues: hoss20120321nofix36

          Show
          Hoss Man added a comment - Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently. email notification suppressed to prevent mass-spam psuedo-unique token identifying these issues: hoss20120321nofix36
          Hide
          Robert Muir added a comment -

          3.4 -> 3.5

          Show
          Robert Muir added a comment - 3.4 -> 3.5
          Hide
          Robert Muir added a comment -

          Bulk move 3.2 -> 3.3

          Show
          Robert Muir added a comment - Bulk move 3.2 -> 3.3
          Hide
          Hoss Man added a comment -

          Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

          http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

          Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

          A unique token for finding these 240 issues in the future: hossversioncleanup20100527

          Show
          Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
          Hide
          Noble Paul added a comment -

          Replicating files may not be useful if the components are not informed about it

          Show
          Noble Paul added a comment - Replicating files may not be useful if the components are not informed about it
          Hide
          Noble Paul added a comment -

          Would it make more sense for the caller to specify (in the request) which files to replicate, thus giving it full control over what to replicate when..

          I suggested one change in the master because it would make the configuration simpler (only one place) and it must cover most of the use cases . We may not need more flexibility unless we have usecases to support this.

          It'd be nice to be able to have various (and custom) triggers that indicate the need and what to replicate.

          currently the trigger is the change in the file's timestamp. So if you wish that your file should be replicated , just 'touch' the file (or overwrite it) and it becomes a candidate for replication.

          Show
          Noble Paul added a comment - Would it make more sense for the caller to specify (in the request) which files to replicate, thus giving it full control over what to replicate when.. I suggested one change in the master because it would make the configuration simpler (only one place) and it must cover most of the use cases . We may not need more flexibility unless we have usecases to support this. It'd be nice to be able to have various (and custom) triggers that indicate the need and what to replicate. currently the trigger is the change in the file's timestamp. So if you wish that your file should be replicated , just 'touch' the file (or overwrite it) and it becomes a candidate for replication.
          Hide
          Erik Hatcher added a comment -

          I've been working with replication a bit lately, and first, this Java-based replication has been a joy to set up and see working.

          One use case I've encountered, replicating when the spell check index has been rebuilt, but the main index remains the same. It'd be nice to be able to have various (and custom) triggers that indicate the need and what to replicate. For example, just touching a query-time synonyms.txt file ought to trigger searchers to pull it (yet this requires a core reload for that file to be reloaded by SynonymFilterFactory, d'oh)

          Show
          Erik Hatcher added a comment - I've been working with replication a bit lately, and first, this Java-based replication has been a joy to set up and see working. One use case I've encountered, replicating when the spell check index has been rebuilt, but the main index remains the same. It'd be nice to be able to have various (and custom) triggers that indicate the need and what to replicate. For example, just touching a query-time synonyms.txt file ought to trigger searchers to pull it (yet this requires a core reload for that file to be reloaded by SynonymFilterFactory, d'oh)
          Hide
          Erik Hatcher added a comment -

          I just want to make a strong note about security here. We must be very careful that replication cannot replicate files outside of solr home or the solr data directory. And perhaps file list selection capability (includes/excludes)... hmmm... sounds a lot like Ant to me .... be centralized between the ShowFileRequestHandler and what can be replicated.

          assert("no way Jose", /replicate?command=sendfile&file=/etc/passwd)

          Show
          Erik Hatcher added a comment - I just want to make a strong note about security here. We must be very careful that replication cannot replicate files outside of solr home or the solr data directory. And perhaps file list selection capability (includes/excludes)... hmmm... sounds a lot like Ant to me .... be centralized between the ShowFileRequestHandler and what can be replicated. assert("no way Jose", /replicate?command=sendfile&file=/etc/passwd)
          Hide
          Otis Gospodnetic added a comment -

          Would it make more sense for the caller to specify (in the request) which files to replicate, thus giving it full control over what to replicate when? Maybe the "realTimeConfFiles" should then not list all conf files that should always be replicated, but instead list all the conf files that are allowed to be replicated when the caller request some of them to be replicated?

          Show
          Otis Gospodnetic added a comment - Would it make more sense for the caller to specify (in the request) which files to replicate, thus giving it full control over what to replicate when? Maybe the "realTimeConfFiles" should then not list all conf files that should always be replicated, but instead list all the conf files that are allowed to be replicated when the caller request some of them to be replicated?
          Hide
          Otis Gospodnetic added a comment -

          From Paul:
          +1

          We should have a separate attributes in the master other than the standard
          <str name="confFiles">a.xml</str>

          say

          <str name="realTimeConfFiles">b.xml</str>

          the files specified in this can be replicated always irrespective of the index

          Show
          Otis Gospodnetic added a comment - From Paul: +1 We should have a separate attributes in the master other than the standard <str name="confFiles">a.xml</str> say <str name="realTimeConfFiles">b.xml</str> the files specified in this can be replicated always irrespective of the index

            People

            • Assignee:
              Unassigned
              Reporter:
              Otis Gospodnetic
            • Votes:
              2 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development