Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9566

Can we avoid doing recovery when collections are first created?

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.3
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      When a core starts up as part of a collection, and it's not a shard leader, it goes into recovery, part of which involves a 7 second wait (set in SOLR-7141) to ensure that updates being sent from the leader don't get missed in between buffering and replication.

      This has the unfortunate side-effect of adding a 7-second pause to collection creation whenever the replication factor is 2 or more, which slows down tests massively - for example, DeleteReplicaTest takes about 54 seconds to execute on my machine, 28 seconds of which is just pauses - over 50% of execution time. It's not actually possible to add documents to a collection before the creation request has returned, so the recovery stage here isn't strictly speaking necessary.

      I think we could try adding a parameter to a CoreAdmin create request that says the core is being created as part of a new collection, so it doesn't need to try and recover from it's leader when it starts up. Does this sound sensible, or am I missing something here?

      1. SOLR-9566.patch
        15 kB
        Alan Woodward

        Activity

        Hide
        romseygeek Alan Woodward added a comment -

        Here's a patch adding a 'newCollection' parameter to the core create command. I haven't run it through a full test yet, but DeleteReplicaTest went from 54 seconds to 20 seconds, which is a very nice speedup.

        Mark Miller Yonik Seeley is this proposal sane?

        Show
        romseygeek Alan Woodward added a comment - Here's a patch adding a 'newCollection' parameter to the core create command. I haven't run it through a full test yet, but DeleteReplicaTest went from 54 seconds to 20 seconds, which is a very nice speedup. Mark Miller Yonik Seeley is this proposal sane?
        Hide
        romseygeek Alan Woodward added a comment -

        I got a verbal +1 from Mark Miller at Revolution, so I'll commit this shortly. Anybody who disagrees should speak up now...

        Show
        romseygeek Alan Woodward added a comment - I got a verbal +1 from Mark Miller at Revolution, so I'll commit this shortly. Anybody who disagrees should speak up now...
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit c9eb9e10e4168ae1fba6d122ab80aa1402f33fce in lucene-solr's branch refs/heads/branch_6x from Alan Woodward
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c9eb9e1 ]

        SOLR-9566: Don't put replicas into recovery when collections are created

        Show
        jira-bot ASF subversion and git services added a comment - Commit c9eb9e10e4168ae1fba6d122ab80aa1402f33fce in lucene-solr's branch refs/heads/branch_6x from Alan Woodward [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c9eb9e1 ] SOLR-9566 : Don't put replicas into recovery when collections are created
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 65f55802ee01b90a7f529de270d5d866a2282a40 in lucene-solr's branch refs/heads/master from Alan Woodward
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=65f5580 ]

        SOLR-9566: Don't put replicas into recovery when collections are created

        Show
        jira-bot ASF subversion and git services added a comment - Commit 65f55802ee01b90a7f529de270d5d866a2282a40 in lucene-solr's branch refs/heads/master from Alan Woodward [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=65f5580 ] SOLR-9566 : Don't put replicas into recovery when collections are created
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.3.0 release.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

          People

          • Assignee:
            romseygeek Alan Woodward
            Reporter:
            romseygeek Alan Woodward
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development