Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13975

ConcurrentUpdateSolrClient connection stall prevention

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 8.3, 8.4
    • 8.4
    • None
    • None

    Description

      When a Solr process, which hosts replicas of a collection, is suspended - that is, the OS process is suspended using eg. kill -STOP <pid> - a long stall may occur in CUSC until a socket timeout is reached.

      During this stall updates from the leader are not forwarded to any replica, even though other replicas are still active and can receive updates. If the sender uses CUSC (eg. via CloudSolrClient) then it becomes stalled because the leader stops processing updates, too.

      This situation is caused by several issues:

      • when a process is suspended its sockets remain open - so there is no immediate disconnect as if the process died, but the process becomes unresponsive. Eventually, a socket timeout will be reached (distribUpdateSoTimeout) - but in the default version of solr.xml this is set to 10 min. During this time all indexing to that shard will be stuck.
      • there are several infinite for loops in CUSC (eg. in blockUntilFinished, waitForEmptyQueue and even in request), which rely either on the relatively quick success of the call or an exception to be thrown. However, in this situation neither happens quickly - the call is stuck waiting for the remote end until soTimeout expires.

      This issue proposes to add a stall prevention logic, which would break these infinite loops long before the socket timeout occurs based on the progress of the queue processing.

      This is a follow-up to SOLR-13896.

      Attachments

        1. SOLR-13975.patch
          23 kB
          Andrzej Bialecki
        2. SOLR-13975.patch
          10 kB
          Andrzej Bialecki

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ab Andrzej Bialecki
            ab Andrzej Bialecki
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment