Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-12833

Use timed-out lock in DistributedUpdateProcessor

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      There is a synchronize block that blocks other update requests whose IDs fall in the same hash bucket. The update waits forever until it gets the lock at the synchronize block, this can be a problem in some cases.

       

      Some add/update requests (for example updates with spatial/shape analysis) like may take time (30+ seconds or even more), this would the request time out and fail.

      Client may retry the same requests multiple times or several minutes, this would make things worse.

      The server side receives all the update requests but all except one can do nothing, have to wait there. This wastes precious memory and cpu resource.

      We have seen the case 2000+ threads are blocking at the synchronize lock, and only a few updates are making progress. Each thread takes 3+ mb memory which causes OOM.

      Also if the update can't get the lock in expected time range, its better to fail fast.

       

      We can have one configuration in solrconfig.xml: updateHandler/versionLock/timeInMill, so users can specify how long they want to wait the version bucket lock.

      The default value can be -1, so it behaves same - wait forever until it gets the lock.

      Attachments

        1. threadDump.txt
          36 kB
          Andrzej Bialecki
        2. SOLR-12833-noint.patch
          5 kB
          Andrzej Bialecki
        3. SOLR-12833.patch
          20 kB
          Mark Miller
        4. SOLR-12833.patch
          19 kB
          Mark Miller

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            markrmiller@gmail.com Mark Miller
            yuanyun.cn jefferyyuan
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                Slack

                  Issue deployment