Solr
  1. Solr
  2. SOLR-624

patch: Don't take snapshot if there are no differences

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.3
    • Component/s: replication (scripts)
    • Labels:
      None

      Description

      This is similar in concept to a change I made several years ago in Solar. Cronned snapshooters can quickly generate a lot of snaps which will then be unnecessarily distributed to the slaves if there hasn't been any changes in that period.

      Adds a "check" argument to make sure there where changes to the index before taking the snap.

      1. solr.check.patch
        0.7 kB
        Richard "Trey" Hyde
      2. solr.check.patch
        0.7 kB
        Richard "Trey" Hyde
      3. SOLR-001.patch
        1 kB
        Richard "Trey" Hyde

        Activity

        Hide
        Richard "Trey" Hyde added a comment -

        Patch to snapshooter

        Show
        Richard "Trey" Hyde added a comment - Patch to snapshooter
        Hide
        Shalin Shekhar Mangar added a comment -

        Just curious, why would you cron snapshooter instead of using the postCommit/postOptimize hook?

        Show
        Shalin Shekhar Mangar added a comment - Just curious, why would you cron snapshooter instead of using the postCommit/postOptimize hook?
        Hide
        Doug Steigerwald added a comment -

        If you have lots of cores committing very often (like we do), it can totally thrash your system. I was planning on implementing something like this at very soon. Glad someone else already did.

        Show
        Doug Steigerwald added a comment - If you have lots of cores committing very often (like we do), it can totally thrash your system. I was planning on implementing something like this at very soon. Glad someone else already did.
        Hide
        Richard "Trey" Hyde added a comment -

        The original patch was before postCommit/postOptimize hooks existed (or at least before I know about them) in the days before Solr was open sourced. Threw this quickly together today when I was annoyed when using snapshooter on the command line (testing MacOS, FreeBSD compatibility among other things).

        It seems there is one other person that might find it useful.

        Show
        Richard "Trey" Hyde added a comment - The original patch was before postCommit/postOptimize hooks existed (or at least before I know about them) in the days before Solr was open sourced. Threw this quickly together today when I was annoyed when using snapshooter on the command line (testing MacOS, FreeBSD compatibility among other things). It seems there is one other person that might find it useful.
        Hide
        Hoss Man added a comment -

        FWIW: issues with the amount of ram/swap needed in order to fork&exec the replication scripts from java is another reason i know some people cron snapshooter.

        short circuiting in the NOOP case out is rarely a bad idea.

        one thing i notice skimming the patch: snapshot.* is used to get the list of all snapshots – didn't we removed similar wildcards from the other scripts because it barfs when the number of snapshots gets really big?

        Show
        Hoss Man added a comment - FWIW: issues with the amount of ram/swap needed in order to fork&exec the replication scripts from java is another reason i know some people cron snapshooter. short circuiting in the NOOP case out is rarely a bad idea. one thing i notice skimming the patch: snapshot.* is used to get the list of all snapshots – didn't we removed similar wildcards from the other scripts because it barfs when the number of snapshots gets really big?
        Hide
        Bill Au added a comment -

        Yes, the '*' will cost problem if there are many snapshots. The snappuller script has a shell command to determine the latest snapshot on the master. We can use the same command in snapshooter.

        Show
        Bill Au added a comment - Yes, the '*' will cost problem if there are many snapshots. The snappuller script has a shell command to determine the latest snapshot on the master. We can use the same command in snapshooter.
        Hide
        Richard "Trey" Hyde added a comment -

        My copy of snapcleaner (-N option) from a few weeks ago in 1.3 is also doing the snap* thing, you might want to look into that one too.

        Show
        Richard "Trey" Hyde added a comment - My copy of snapcleaner (-N option) from a few weeks ago in 1.3 is also doing the snap* thing, you might want to look into that one too.
        Hide
        Richard "Trey" Hyde added a comment -

        Updated patch resolves issues with wildcards with too many matches.

        Show
        Richard "Trey" Hyde added a comment - Updated patch resolves issues with wildcards with too many matches.
        Hide
        Grant Ingersoll added a comment -

        Richard,

        Can you please generate the patch per the instructions at http://wiki.apache.org/solr/HowToContribute

        Thanks,
        Grant

        Show
        Grant Ingersoll added a comment - Richard, Can you please generate the patch per the instructions at http://wiki.apache.org/solr/HowToContribute Thanks, Grant
        Hide
        Richard "Trey" Hyde added a comment -

        Patch according to the Apache guidelines.

        Show
        Richard "Trey" Hyde added a comment - Patch according to the Apache guidelines.
        Hide
        Grant Ingersoll added a comment -

        Committed revision 683785.

        Show
        Grant Ingersoll added a comment - Committed revision 683785.

          People

          • Assignee:
            Grant Ingersoll
            Reporter:
            Richard "Trey" Hyde
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development