Solr
  1. Solr
  2. SOLR-8203

Stop processing updates more quickly on shutdown

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.4
    • Component/s: None
    • Labels:
      None

      Description

      As we discovered in SOLR-8129, existing update streams can continue to be processed for some time when the CoreContainer is being shut down. If this node is the leader, updates can continue to flow to replicas over the existing streaming client (although new or idle clients won't be allowed to send anything).

      This can cause large reorders of updates and shard inconsistencies that we can't detect with the current PeerSync mechanism.

      1. SOLR-8203.patch
        0.8 kB
        Yonik Seeley

        Issue Links

          Activity

          Hide
          Yonik Seeley added a comment -

          Here's the patch I'm currently testing with.
          It only checks when in ZK mode... it was easier to add it in a single place, and shouldn't really matter in other modes.

          Show
          Yonik Seeley added a comment - Here's the patch I'm currently testing with. It only checks when in ZK mode... it was easier to add it in a single place, and shouldn't really matter in other modes.
          Hide
          Yonik Seeley added a comment -

          I looped the test overnight (adds only) and there were no shard inconsistency type fails. Although there were plenty of other fails (50 in fact), this is the first time that there was no shard inconsistency.

          Mark has been testing "interrupt the update executor on shutdown" and that also looks like it fixes the issue. That will also have the effect of stopping other operations more quickly (i.e. requests to other nodes that use that update executor), and in general it seems like a good thing to stop more quickly. Should that be added here?

          Show
          Yonik Seeley added a comment - I looped the test overnight (adds only) and there were no shard inconsistency type fails. Although there were plenty of other fails (50 in fact), this is the first time that there was no shard inconsistency. Mark has been testing "interrupt the update executor on shutdown" and that also looks like it fixes the issue. That will also have the effect of stopping other operations more quickly (i.e. requests to other nodes that use that update executor), and in general it seems like a good thing to stop more quickly. Should that be added here?
          Hide
          ASF subversion and git services added a comment -

          Commit 1712034 from Yonik Seeley in branch 'dev/trunk'
          [ https://svn.apache.org/r1712034 ]

          SOLR-8203: reject updates if core container is shutting down

          Show
          ASF subversion and git services added a comment - Commit 1712034 from Yonik Seeley in branch 'dev/trunk' [ https://svn.apache.org/r1712034 ] SOLR-8203 : reject updates if core container is shutting down
          Hide
          ASF subversion and git services added a comment -

          Commit 1712035 from Yonik Seeley in branch 'dev/trunk'
          [ https://svn.apache.org/r1712035 ]

          SOLR-8203: reject updates if core container is shutting down

          Show
          ASF subversion and git services added a comment - Commit 1712035 from Yonik Seeley in branch 'dev/trunk' [ https://svn.apache.org/r1712035 ] SOLR-8203 : reject updates if core container is shutting down
          Hide
          ASF subversion and git services added a comment -

          Commit 1712036 from Yonik Seeley in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1712036 ]

          SOLR-8203: reject updates if core container is shutting down

          Show
          ASF subversion and git services added a comment - Commit 1712036 from Yonik Seeley in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1712036 ] SOLR-8203 : reject updates if core container is shutting down
          Hide
          Mark Miller added a comment -

          Should that be added here?

          +1 - I think we should do this too. We can add some doc around the fact that the update executor should not do any disk IO. Technically, it's probably fine since it lives the length of the container, but I think it's best to avoid it for proper shutdown.

          Show
          Mark Miller added a comment - Should that be added here? +1 - I think we should do this too. We can add some doc around the fact that the update executor should not do any disk IO. Technically, it's probably fine since it lives the length of the container, but I think it's best to avoid it for proper shutdown.
          Hide
          ASF subversion and git services added a comment -

          Commit 1712111 from Yonik Seeley in branch 'dev/trunk'
          [ https://svn.apache.org/r1712111 ]

          SOLR-8203: use shutdownNow on updateExecutor

          Show
          ASF subversion and git services added a comment - Commit 1712111 from Yonik Seeley in branch 'dev/trunk' [ https://svn.apache.org/r1712111 ] SOLR-8203 : use shutdownNow on updateExecutor
          Hide
          ASF subversion and git services added a comment -

          Commit 1712112 from Yonik Seeley in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1712112 ]

          SOLR-8203: use shutdownNow on updateExecutor

          Show
          ASF subversion and git services added a comment - Commit 1712112 from Yonik Seeley in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1712112 ] SOLR-8203 : use shutdownNow on updateExecutor

            People

            • Assignee:
              Yonik Seeley
              Reporter:
              Yonik Seeley
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development