Solr
  1. Solr
  2. SOLR-433

MultiCore and SpellChecker replication

    Details

      Description

      With MultiCore functionality coming along, it looks like we'll need to be able to:
      A) snapshot each core's index directory, and
      B) replicate any and all cores' complete data directories, not just their index directories.

      Pulled from the "spellchecker and multi-core index replication" thread - http://markmail.org/message/pj2rjzegifd6zm7m

      Otis:

      I think that makes sense - distribute everything for a given core, not just its index. And the spellchecker could then also have its data dir (and only index/ underneath really) and be replicated in the same fashion.

      Right?

      Ryan:

      Yes, that was my thought. If an arbitrary directory could be distributed, then you could have

      /path/to/dist/index/...
      /path/to/dist/spelling-index/...
      /path/to/dist/foo

      and that would all get put into a snapshot. This would also let you put multiple cores within a single distribution:

      /path/to/dist/core0/index/...
      /path/to/dist/core0/spelling-index/...
      /path/to/dist/core0/foo
      /path/to/dist/core1/index/...
      /path/to/dist/core1/spelling-index/...
      /path/to/dist/core1/foo

      1. spellindexfix.patch
        0.8 kB
        Doug Steigerwald
      2. SOLR-433-r698590.patch
        17 kB
        Nicolas Lalevée
      3. SOLR-433.patch
        16 kB
        Jonathan Lee
      4. SOLR-433.patch
        17 kB
        Jonathan Lee
      5. SOLR-433.patch
        18 kB
        Chris Haggstrom
      6. SOLR-433.patch
        19 kB
        Jeremy Hinegardner
      7. solr-433.patch
        13 kB
        Doug Steigerwald
      8. SOLR-433_unified.patch
        13 kB
        Jeremy Hinegardner
      9. RunExecutableListener.patch
        0.9 kB
        Doug Steigerwald

        Issue Links

          Activity

          Hide
          Otis Gospodnetic added a comment -

          I think this is no longer needed, since spellchecker no longer needs a separate index. Resolving as Won't Fix.

          Show
          Otis Gospodnetic added a comment - I think this is no longer needed, since spellchecker no longer needs a separate index. Resolving as Won't Fix.
          Hide
          Hoss Man added a comment -

          Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently.

          email notification suppressed to prevent mass-spam
          psuedo-unique token identifying these issues: hoss20120321nofix36

          Show
          Hoss Man added a comment - Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently. email notification suppressed to prevent mass-spam psuedo-unique token identifying these issues: hoss20120321nofix36
          Hide
          Robert Muir added a comment -

          3.4 -> 3.5

          Show
          Robert Muir added a comment - 3.4 -> 3.5
          Hide
          Robert Muir added a comment -

          Bulk move 3.2 -> 3.3

          Show
          Robert Muir added a comment - Bulk move 3.2 -> 3.3
          Hide
          Hoss Man added a comment -

          Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

          http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

          Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

          A unique token for finding these 240 issues in the future: hossversioncleanup20100527

          Show
          Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
          Hide
          Jason Rutherglen added a comment -

          Are the existing patches for multiple cores or only for spellchecking?

          Show
          Jason Rutherglen added a comment - Are the existing patches for multiple cores or only for spellchecking?
          Hide
          Jeremy Hinegardner added a comment -

          I've updated the patch to apply cleanly against trunk.

          Show
          Jeremy Hinegardner added a comment - I've updated the patch to apply cleanly against trunk.
          Hide
          Shalin Shekhar Mangar added a comment -

          Marking for 1.5

          Show
          Shalin Shekhar Mangar added a comment - Marking for 1.5
          Hide
          Chris Haggstrom added a comment -

          I've been using the patch submitted by Jonathan Lee on 10-02-08 for replicating a spelling directory in addition to the index, and it works very well for that purpose.

          I'm attaching a slightly modified patch that allows the snapshooter "-c" option to work with an index or spelling directory that is not named "index".

          Show
          Chris Haggstrom added a comment - I've been using the patch submitted by Jonathan Lee on 10-02-08 for replicating a spelling directory in addition to the index, and it works very well for that purpose. I'm attaching a slightly modified patch that allows the snapshooter "-c" option to work with an index or spelling directory that is not named "index".
          Hide
          Jonathan Lee added a comment -

          Includes Stephane's fixes for snappuller & snapinstaller and some minor edits

          Show
          Jonathan Lee added a comment - Includes Stephane's fixes for snappuller & snapinstaller and some minor edits
          Hide
          Stephane Bailliez added a comment -

          Note that on the last patches, the grep to retrieve the snapshot is incorrect:

          ls ${data_dir}|grep "${snap_prefix}\."|grep -v wip|sort -r|head -1
          

          would always retrieve the latest one on the ls, it needs to be with an anchor in the grep for the prefix otherwise it will never update the index snapshot (since 'snapshot' is present in every snapshot of index)

          ls ${data_dir}|grep "^${snap_prefix}\."|grep -v wip|sort -r|head -1
          

          should be changed in snappuller and snapinstaller

          Show
          Stephane Bailliez added a comment - Note that on the last patches, the grep to retrieve the snapshot is incorrect: ls ${data_dir}|grep "${snap_prefix}\."|grep -v wip|sort -r|head -1 would always retrieve the latest one on the ls, it needs to be with an anchor in the grep for the prefix otherwise it will never update the index snapshot (since 'snapshot' is present in every snapshot of index) ls ${data_dir}|grep "^${snap_prefix}\."|grep -v wip|sort -r|head -1 should be changed in snappuller and snapinstaller
          Hide
          Nicolas Lalevée added a comment -

          The patch that works the best for me is the last one, Jonathan's one, as I run Solr with only one core but with two spellchecker indexes.

          • I have fixed in that patch the snapcleaner when called to explicitely clean the main index (./snapcleaner -i index).
          • I also fixed every "USAGE" printing.
          • and I have included the patch on RunExecutableListener and a little simplified it (it can now access to the solr core instance).
          Show
          Nicolas Lalevée added a comment - The patch that works the best for me is the last one, Jonathan's one, as I run Solr with only one core but with two spellchecker indexes. I have fixed in that patch the snapcleaner when called to explicitely clean the main index ( ./snapcleaner -i index ). I also fixed every "USAGE" printing. and I have included the patch on RunExecutableListener and a little simplified it (it can now access to the solr core instance).
          Show
          Jonathan Lee added a comment - (patch from comment https://issues.apache.org/jira/browse/SOLR-433?focusedCommentId=12620430#action_12620430 )
          Hide
          Jonathan Lee added a comment - - edited

          I'm interested in using this patch to replicate the spell index, but I am not using multiple cores. The scripts in the patch do not work for single core setups since it makes an assumptions that the core name will appear in the rsync path.

          I am attaching another version of the patch which makes a few changes:

          • snapcleaner, snapshooter, and snappuller are backwards compatible with the naming convention used by the current scripts
          • Fixed a few small bugs where static strings were used instead of the corresponding variables
          • Avoid assuming that an executable will accept a '-c' parameter to specify the core name. Instead, allow a RunExecutableListener to be conditionally executed for specified cores. RunExecutableListener now accepts a <arr name="cores"> parameter. You would need to add a listener for each core. For example:
            <listener event="postOptimize" class="core.RunExecutableListener">
                <arr name="cores"> <str>core0</str> </arr>
                <str name="exe">solr/bin/snapshooter</str>
                <str name="dir">.</str>
                <arr name="args"> <str>-d /usr/local/solr/core0/data</str> </arr>
            </listener>
            <listener event="postOptimize" class="core.RunExecutableListener">
                <arr name="cores"> <str>core1</str> </arr>
                <str name="exe">solr/bin/snapshooter</str>
                <str name="dir">.</str>
                <arr name="args"> <str>-d /usr/local/solr/core1/data</str> </arr>
            </listener>
            

            (Another reasonable alternative to this might also be to accept variables like $

            {SOLR_CORE}

            in <args> and <env> which are resolved by RunExecutableListener.)

          • Removed the '-c' parameter from snappuller, replacing it instead with the '-r' option to specify an rsync module. This allows us to not assume the location of the core's data path. Instead we would add a new module to rsyncd.conf for each core. e.g.
            ...
            [core0]
                path = /usr/local/solr/core0/data
                comment = core0
            [core1]
                path = /usr/local/solr/core1/data
                comment = core1
            ...
            

            then use:

            ./snappuller -r core0
            
          Show
          Jonathan Lee added a comment - - edited I'm interested in using this patch to replicate the spell index, but I am not using multiple cores. The scripts in the patch do not work for single core setups since it makes an assumptions that the core name will appear in the rsync path. I am attaching another version of the patch which makes a few changes: snapcleaner, snapshooter, and snappuller are backwards compatible with the naming convention used by the current scripts Fixed a few small bugs where static strings were used instead of the corresponding variables Avoid assuming that an executable will accept a '-c' parameter to specify the core name. Instead, allow a RunExecutableListener to be conditionally executed for specified cores. RunExecutableListener now accepts a <arr name="cores"> parameter. You would need to add a listener for each core. For example: <listener event= "postOptimize" class= "core.RunExecutableListener" > <arr name= "cores" > <str>core0</str> </arr> <str name= "exe" >solr/bin/snapshooter</str> <str name= "dir" >.</str> <arr name= "args" > <str>-d /usr/local/solr/core0/data</str> </arr> </listener> <listener event= "postOptimize" class= "core.RunExecutableListener" > <arr name= "cores" > <str>core1</str> </arr> <str name= "exe" >solr/bin/snapshooter</str> <str name= "dir" >.</str> <arr name= "args" > <str>-d /usr/local/solr/core1/data</str> </arr> </listener> (Another reasonable alternative to this might also be to accept variables like $ {SOLR_CORE} in <args> and <env> which are resolved by RunExecutableListener.) Removed the '-c' parameter from snappuller, replacing it instead with the '-r' option to specify an rsync module. This allows us to not assume the location of the core's data path. Instead we would add a new module to rsyncd.conf for each core. e.g. ... [core0] path = /usr/local/solr/core0/data comment = core0 [core1] path = /usr/local/solr/core1/data comment = core1 ... then use: ./snappuller -r core0
          Hide
          Jeremy Hinegardner added a comment - - edited

          I've combined solr-433.patch and RunExecutableListener.patch into a single patch against svn trunk. This also fixes a syntax bug in RunExecutableListener where super(core); was not invoked.

          Show
          Jeremy Hinegardner added a comment - - edited I've combined solr-433.patch and RunExecutableListener.patch into a single patch against svn trunk. This also fixes a syntax bug in RunExecutableListener where super(core); was not invoked.
          Hide
          Jeremy Hinegardner added a comment -

          If this patch works for folks, I would like to see it committed and put in the nightly snapshot. Or at least the RunExecutableListener.patch and solr-433.patch.

          If there is any more work here, I'd be happy to work on it.

          Show
          Jeremy Hinegardner added a comment - If this patch works for folks, I would like to see it committed and put in the nightly snapshot. Or at least the RunExecutableListener.patch and solr-433.patch. If there is any more work here, I'd be happy to work on it.
          Hide
          Otis Gospodnetic added a comment -

          You definitely want to put this in a separate JIRA issue and in Lucene's JIRA where you'll find a few similar issues already.

          Show
          Otis Gospodnetic added a comment - You definitely want to put this in a separate JIRA issue and in Lucene's JIRA where you'll find a few similar issues already.
          Hide
          Doug Steigerwald added a comment -

          This may be better suited as another bug, or even posted in the lucene project, but I also made a small patch for the lucene spellchecker.

          Hitting the spellcheck request handler with a reopen command wasn't working after we installed a new spell index snapshot. Searcher was being created for the new index, but not a reader.

          Also, if you rebuild the spell index after it has already been built, it cleans out the index. You then have to send it a rebuild again to actually rebuild the index. Frequency of words in the spell index seemed to remain constant when rebuilding the spell index multiple times.

          Show
          Doug Steigerwald added a comment - This may be better suited as another bug, or even posted in the lucene project, but I also made a small patch for the lucene spellchecker. Hitting the spellcheck request handler with a reopen command wasn't working after we installed a new spell index snapshot. Searcher was being created for the new index, but not a reader. Also, if you rebuild the spell index after it has already been built, it cleans out the index. You then have to send it a rebuild again to actually rebuild the index. Frequency of words in the spell index seemed to remain constant when rebuilding the spell index multiple times.
          Hide
          Doug Steigerwald added a comment -

          Should work. Had to pull changes out of a separate class I write to create this patch.

          Show
          Doug Steigerwald added a comment - Should work. Had to pull changes out of a separate class I write to create this patch.
          Hide
          Doug Steigerwald added a comment - - edited

          We have a ctl script that controls all of the functions and makes sure we don't run some things on the slaves (ie, snappuller, snapinstaller, rsyncd stuff).

          We pass it a core and an index type:

          ./snapctl -a [rsyncd-enable/snappuller/snapinstaller/etc] -c core0 -i spell
          

          spell is the name of our spellcheck index. Default index is 'index'.

          This is testing well in QA right now. Hopefully this will help others out and maybe we'll have something similar committed soon.

          Here's the basics of our snapctl script. It's really tailored to our environment, so I probably won't post it as is.

          $SOLR_BIN=/home/dsteiger/apps/solr/solr/bin
          $CORES_PATH=/home/dsteiger/local/solr/cores
          $CORE=core0 # from -c arg, default is 'null'
          $INDEX=spell  # from -i arg, default is 'index'
          $SOLR_LOGS=/home/dsteiger/apps/solr/logs # symlinked to somewhere else
          # $MASTER_HOST is determined based on environment (devel/qa/prod) from the scripts.conf
          
          $SOLR_BIN/rsyncd-enable
          
          $SOLR_BIN/rsyncd-disable
          
          $SOLR_BIN/rsyncd-start -d $CORES_PATH
          
          $SOLR_BIN/rsyncd-stop
          
          $SOLR_BIN/snappuller-enable
          
          $SOLR_BIN/snappuller-disable
          
          $SOLR_BIN/snapshooter -d $CORES_PATH/$CORE/data -i $INDEX
          
          $SOLR_BIN/snappuller -M $MASTER_HOST -S $SOLR_LOGS/clients -D $CORES_PATH/$CORE/data -d $CORES_PATH/$CORE/data -z -c $CORE -i $INDEX
          
          $SOLR_BIN/snapinstaller -M $MASTER_HOST -S $SOLR_LOGS/clients -d $CORES_PATH/$CORE/data -c $CORE -i $INDEX
          
          $SOLR_BIN/snapcleaner -D 1 -d $CORES_PATH/$CORE/data -i $INDEX
          

          We also modified core.RunExecutableListener to be able to pass the core name to out snapctl script.

              <listener event="postCommit" class="core.RunExecutableListener">
                  <str name="exe">./solr/bin/snapctl</str>
                  <str name="dir">.</str>
                  <bool name="wait">true</bool>
                  <bool name="coreName">true</bool>
                  <arr name="args"><str>-a snapshooter</str><str>-i index</str></arr>
              </listener>
              <listener event="postOptimize" class="core.RunExecutableListener">
                  <str name="exe">./solr/bin/snapctl</str>
                  <str name="dir">.</str>
                  <bool name="wait">true</bool>
                  <bool name="coreName">true</bool>
                  <arr name="args"> <str>-a snapshooter</str> </arr>
              </listener>
          

          Going to attach patch to RunExecutableListener we're using.

          Show
          Doug Steigerwald added a comment - - edited We have a ctl script that controls all of the functions and makes sure we don't run some things on the slaves (ie, snappuller, snapinstaller, rsyncd stuff). We pass it a core and an index type: ./snapctl -a [rsyncd-enable/snappuller/snapinstaller/etc] -c core0 -i spell spell is the name of our spellcheck index. Default index is 'index'. This is testing well in QA right now. Hopefully this will help others out and maybe we'll have something similar committed soon. Here's the basics of our snapctl script. It's really tailored to our environment, so I probably won't post it as is. $SOLR_BIN=/home/dsteiger/apps/solr/solr/bin $CORES_PATH=/home/dsteiger/local/solr/cores $CORE=core0 # from -c arg, default is 'null' $INDEX=spell # from -i arg, default is 'index' $SOLR_LOGS=/home/dsteiger/apps/solr/logs # symlinked to somewhere else # $MASTER_HOST is determined based on environment (devel/qa/prod) from the scripts.conf $SOLR_BIN/rsyncd-enable $SOLR_BIN/rsyncd-disable $SOLR_BIN/rsyncd-start -d $CORES_PATH $SOLR_BIN/rsyncd-stop $SOLR_BIN/snappuller-enable $SOLR_BIN/snappuller-disable $SOLR_BIN/snapshooter -d $CORES_PATH/$CORE/data -i $INDEX $SOLR_BIN/snappuller -M $MASTER_HOST -S $SOLR_LOGS/clients -D $CORES_PATH/$CORE/data -d $CORES_PATH/$CORE/data -z -c $CORE -i $INDEX $SOLR_BIN/snapinstaller -M $MASTER_HOST -S $SOLR_LOGS/clients -d $CORES_PATH/$CORE/data -c $CORE -i $INDEX $SOLR_BIN/snapcleaner -D 1 -d $CORES_PATH/$CORE/data -i $INDEX We also modified core.RunExecutableListener to be able to pass the core name to out snapctl script. <listener event= "postCommit" class= "core.RunExecutableListener" > <str name= "exe" > ./solr/bin/snapctl </str> <str name= "dir" > . </str> <bool name= "wait" > true </bool> <bool name= "coreName" > true </bool> <arr name= "args" > <str> -a snapshooter </str> <str> -i index </str> </arr> </listener> <listener event= "postOptimize" class= "core.RunExecutableListener" > <str name= "exe" > ./solr/bin/snapctl </str> <str name= "dir" > . </str> <bool name= "wait" > true </bool> <bool name= "coreName" > true </bool> <arr name= "args" > <str> -a snapshooter </str> </arr> </listener> Going to attach patch to RunExecutableListener we're using.
          Hide
          Hoss Man added a comment -

          as i recall, most of the scripts currently take a "-d data_dir" option ... and then use makethe following assumptions...
          1) $

          {data_dir}/index is what gets snapshooted/snapinstalled
          2) ${data_dir}

          /snapshot* is the pattern for naming snapshots.

          a good way to evolve the scripts would probably be to have an alternate set of options ... maybe "-D dir" and "-S snapshots" that assumes $

          {dir}

          is where all the data to be snapshooted/snapinstalled lives, and $

          {snapshots}

          is where all snapshoots live.

          (except i think some of the scripts already have a -D .. snappuller or snapcleaner maybe?)

          alternate idea: we don't have to add a command line option at all ... support for something like this could require special scripts.conf options.

          Show
          Hoss Man added a comment - as i recall, most of the scripts currently take a "-d data_dir" option ... and then use makethe following assumptions... 1) $ {data_dir}/index is what gets snapshooted/snapinstalled 2) ${data_dir} /snapshot* is the pattern for naming snapshots. a good way to evolve the scripts would probably be to have an alternate set of options ... maybe "-D dir" and "-S snapshots" that assumes $ {dir} is where all the data to be snapshooted/snapinstalled lives, and $ {snapshots} is where all snapshoots live. (except i think some of the scripts already have a -D .. snappuller or snapcleaner maybe?) alternate idea: we don't have to add a command line option at all ... support for something like this could require special scripts.conf options.
          Hide
          Doug Steigerwald added a comment -

          It's on our TODO list here if no one was working on it yet. I'll see what we can get done based on the comments in the description.

          Show
          Doug Steigerwald added a comment - It's on our TODO list here if no one was working on it yet. I'll see what we can get done based on the comments in the description.
          Hide
          Otis Gospodnetic added a comment -

          I don't think so, Doug. If you are asking if you should start working on this and contribute a patch, by all means, please!

          Show
          Otis Gospodnetic added a comment - I don't think so, Doug. If you are asking if you should start working on this and contribute a patch, by all means, please!
          Hide
          Doug Steigerwald added a comment -

          Is anyone looking into this?

          Show
          Doug Steigerwald added a comment - Is anyone looking into this?

            People

            • Assignee:
              Unassigned
              Reporter:
              Otis Gospodnetic
            • Votes:
              10 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development