Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9707

DeleteByQuery forward requests to down replicas and set it in LiR

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4
    • Component/s: SolrCloud
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
    • Flags:
      Patch

      Description

      DeleteByQuery, unlike other requests, does not filter out the down replicas. Thus, the update is still forwarded to the down replica and fails, and the leader then sets the replica in LiR. In a cluster where there are lots of deleteByQuery requests, this can flood the /overseer/queue.

      1. SOLR-9707.diff
        0.9 kB
        Jessica Cheng Mallet
      2. SOLR-9707.patch
        7 kB
        Varun Thacker

        Activity

        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Yonik Seeley, any idea if that was on purpose to avoid a state race or something?

        Show
        markrmiller@gmail.com Mark Miller added a comment - Yonik Seeley , any idea if that was on purpose to avoid a state race or something?
        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        I don't think so...
        seems like it should follow the same procedure as any other update.

        Show
        yseeley@gmail.com Yonik Seeley added a comment - I don't think so... seems like it should follow the same procedure as any other update.
        Hide
        varunthacker Varun Thacker added a comment -

        Hi Jessica,

        I was trying to write a test case for this and I wasn't able to get it to fail without the patch. I realized that it's because we already filter out replicas which aren't on the live node list ( https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java#L805 )

        Attaching the updated patch with the test case for reference.

        Show
        varunthacker Varun Thacker added a comment - Hi Jessica, I was trying to write a test case for this and I wasn't able to get it to fail without the patch. I realized that it's because we already filter out replicas which aren't on the live node list ( https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java#L805 ) Attaching the updated patch with the test case for reference.
        Hide
        varunthacker Varun Thacker added a comment -

        Its still a good idea to filter out down replica before sending the update. There might be edge cases / bugs which could lead to a replica being in down state even on a live node.

        The tests pass with Jessica's original patch.
        Mark Miller or Yonik Seeley any suggestions on how we can add a test case for this or is it just safe to commit the patch without an explicit test case for it?

        Show
        varunthacker Varun Thacker added a comment - Its still a good idea to filter out down replica before sending the update. There might be edge cases / bugs which could lead to a replica being in down state even on a live node. The tests pass with Jessica's original patch. Mark Miller or Yonik Seeley any suggestions on how we can add a test case for this or is it just safe to commit the patch without an explicit test case for it?
        Hide
        varunthacker Varun Thacker added a comment -

        I'm going to run a couple of more rounds of tests and then commit the patch without the test case.

        Show
        varunthacker Varun Thacker added a comment - I'm going to run a couple of more rounds of tests and then commit the patch without the test case.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit fecbbe081fd4a777f01517fdd8631e69797def38 in lucene-solr's branch refs/heads/master from Varun Thacker
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fecbbe0 ]

        SOLR-9707: Don't forward DeleteByQuery requests to down replicas

        Show
        jira-bot ASF subversion and git services added a comment - Commit fecbbe081fd4a777f01517fdd8631e69797def38 in lucene-solr's branch refs/heads/master from Varun Thacker [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fecbbe0 ] SOLR-9707 : Don't forward DeleteByQuery requests to down replicas
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a243f20973aad98f1961f48b197c90d2877704d4 in lucene-solr's branch refs/heads/branch_6x from Varun Thacker
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a243f20 ]

        SOLR-9707: Don't forward DeleteByQuery requests to down replicas

        Show
        jira-bot ASF subversion and git services added a comment - Commit a243f20973aad98f1961f48b197c90d2877704d4 in lucene-solr's branch refs/heads/branch_6x from Varun Thacker [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a243f20 ] SOLR-9707 : Don't forward DeleteByQuery requests to down replicas
        Hide
        varunthacker Varun Thacker added a comment -

        Thanks Jessica for catching this!

        Show
        varunthacker Varun Thacker added a comment - Thanks Jessica for catching this!
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit fecbbe081fd4a777f01517fdd8631e69797def38 in lucene-solr's branch refs/heads/feature/metrics from Varun Thacker
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fecbbe0 ]

        SOLR-9707: Don't forward DeleteByQuery requests to down replicas

        Show
        jira-bot ASF subversion and git services added a comment - Commit fecbbe081fd4a777f01517fdd8631e69797def38 in lucene-solr's branch refs/heads/feature/metrics from Varun Thacker [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fecbbe0 ] SOLR-9707 : Don't forward DeleteByQuery requests to down replicas

          People

          • Assignee:
            varunthacker Varun Thacker
            Reporter:
            mewmewball Jessica Cheng Mallet
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development