Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9116

Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.5.2, 5.6, 6.0.1, 6.1, 7.0
    • Component/s: None
    • Labels:
    • Flags:
      Patch

      Description

      Fix a leak of SolrIndexSearcher when a SolrCore is reloaded. Added a test to expose this leak when run in many iterations (pretty reliable failure with iters=1K), which passes with the fix (ran iters=10K twice).

      The fundamental issue is that when an invocation of SolrCore#openNewSearcher is racing with SolrCore#close, if this synchronized block (https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1611) in openNewSearcher doesn't check for whether or not the core is closed, it can possibly run after the core runs closeSearcher and assign the newly constructed searcher to realtimeSearcher again, which will never be cleaned up. The fix is to check if the SolrCore is closed inside the synchronized block, and if so, clean up the newly constructed searcher and throw an Exception.

      1. SOLR-9116.patch
        5 kB
        Jessica Cheng Mallet

        Issue Links

          Activity

          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Nice catch!

          +1 LGTM

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Nice catch! +1 LGTM
          Hide
          erickerickson Erick Erickson added a comment -

          I'll commit this in a day or two unless someone beats me to it. I'm traveling so can't quite promise when so if someone really wants to take it please do.

          I confirmed this on 6.x both with and without the actual fix.

          Show
          erickerickson Erick Erickson added a comment - I'll commit this in a day or two unless someone beats me to it. I'm traveling so can't quite promise when so if someone really wants to take it please do. I confirmed this on 6.x both with and without the actual fix.
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          I ran the tests with and without the fix with a 1000 iterations and I encountered failures without the patch but none with the patch. I'll commit this now.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - I ran the tests with and without the fix with a 1000 iterations and I encountered failures without the patch but none with the patch. I'll commit this now.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d6a5c5a539384ba89456de9bbc50ff90ff4fd6ac in lucene-solr's branch refs/heads/master from Shalin Shekhar Mangar
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d6a5c5a ]

          SOLR-9116: Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded

          Show
          jira-bot ASF subversion and git services added a comment - Commit d6a5c5a539384ba89456de9bbc50ff90ff4fd6ac in lucene-solr's branch refs/heads/master from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d6a5c5a ] SOLR-9116 : Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 2dff2b8070ae83e45410b20558ae7820653ffefb in lucene-solr's branch refs/heads/branch_6x from Shalin Shekhar Mangar
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2dff2b8 ]

          SOLR-9116: Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded
          (cherry picked from commit d6a5c5a)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 2dff2b8070ae83e45410b20558ae7820653ffefb in lucene-solr's branch refs/heads/branch_6x from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2dff2b8 ] SOLR-9116 : Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded (cherry picked from commit d6a5c5a)
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Thanks Erick, I took care of this one. Happy travels!

          Thanks Jessica for the fix!

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Thanks Erick, I took care of this one. Happy travels! Thanks Jessica for the fix!
          Hide
          steve_rowe Steve Rowe added a comment -

          Reopening to backport to 6.0.1.

          Show
          steve_rowe Steve Rowe added a comment - Reopening to backport to 6.0.1.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5c859404289bfb0c9a5d7d1df2c07aabf61865c4 in lucene-solr's branch refs/heads/branch_6_0 from Shalin Shekhar Mangar
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5c85940 ]

          SOLR-9116: Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded
          (cherry picked from commit d6a5c5a)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5c859404289bfb0c9a5d7d1df2c07aabf61865c4 in lucene-solr's branch refs/heads/branch_6_0 from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5c85940 ] SOLR-9116 : Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded (cherry picked from commit d6a5c5a)
          Hide
          steve_rowe Steve Rowe added a comment -

          Bulk close issues included in the 6.0.1 release.

          Show
          steve_rowe Steve Rowe added a comment - Bulk close issues included in the 6.0.1 release.
          Hide
          steve_rowe Steve Rowe added a comment -

          Reopening to backport to 5.6 and 5.5.2.

          Show
          steve_rowe Steve Rowe added a comment - Reopening to backport to 5.6 and 5.5.2.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e86eb9d1ecd6f63b15068b368819e31c4c3e1203 in lucene-solr's branch refs/heads/branch_5_5 from Shalin Shekhar Mangar
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e86eb9d ]

          SOLR-9116: Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded
          (cherry picked from commit d6a5c5a)

          Show
          jira-bot ASF subversion and git services added a comment - Commit e86eb9d1ecd6f63b15068b368819e31c4c3e1203 in lucene-solr's branch refs/heads/branch_5_5 from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e86eb9d ] SOLR-9116 : Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded (cherry picked from commit d6a5c5a)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d58bff079cca06fa7d54ced6d73fa7ffb8e0df7f in lucene-solr's branch refs/heads/branch_5x from Shalin Shekhar Mangar
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d58bff0 ]

          SOLR-9116: Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded
          (cherry picked from commit d6a5c5a)

          Show
          jira-bot ASF subversion and git services added a comment - Commit d58bff079cca06fa7d54ced6d73fa7ffb8e0df7f in lucene-solr's branch refs/heads/branch_5x from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d58bff0 ] SOLR-9116 : Race condition causing occasional SolrIndexSearcher leak when SolrCore is reloaded (cherry picked from commit d6a5c5a)
          Hide
          steve_rowe Steve Rowe added a comment -

          Bulk close issues released with 5.5.2.

          Show
          steve_rowe Steve Rowe added a comment - Bulk close issues released with 5.5.2.

            People

            • Assignee:
              shalinmangar Shalin Shekhar Mangar
              Reporter:
              mewmewball Jessica Cheng Mallet
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development