Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6660 Improve the usability for the new Suggester
  3. SOLR-6246

Core fails to reload when AnalyzingInfixSuggester is used as a Suggester

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.8, 4.8.1, 4.9, 5.0, 5.1, 5.2, 5.3, 5.4
    • Fix Version/s: 6.4.1, 6.5, 7.0
    • Labels:
      None

      Description

      LUCENE-5477 - added near-real-time suggest building to AnalyzingInfixSuggester. One of the changes that went in was a writer is persisted now to support real time updates via the add() and update() methods.

      When we call Solr's reload command, a new instance of AnalyzingInfixSuggester is created. When trying to create a new writer on the same Directory a lock cannot be obtained and Solr fails to reload the core.

      Also when AnalyzingInfixLookupFactory throws a RuntimeException we should pass along the original message.

      I am not sure what should be the approach to fix it. Should we have a reloadHook where we close the writer?

      1. SOLR-6246.patch
        19 kB
        Steve Rowe
      2. SOLR-6246.patch
        11 kB
        Varun Thacker
      3. SOLR-6246-test.patch
        13 kB
        Steve Rowe
      4. SOLR-6246-test.patch
        2 kB
        Steve Rowe
      5. SOLR-6246-test.patch
        17 kB
        Varun Thacker
      6. SOLR-6246-test.patch
        12 kB
        Varun Thacker

        Issue Links

          Activity

          Hide
          varunthacker Varun Thacker added a comment -

          When we call Solr's reload command, a new instance of AnalyzingInfixSuggester is created. When trying to create a new writer on the same Directory a lock cannot be obtained and Solr fails to reload the core.

          SolrSuggester adds close hooks but that doesn't work in this case. The reason is that the close hooks gets called only after the new core gets created. CoreContainer.reload() first creates the new core ( which fails ) and only after it has registered the new core does it clean up the old core ( registerCore() )

          We need to clean up the resources before the new core is created or use the same resources again if it hasn't changed.

          Show
          varunthacker Varun Thacker added a comment - When we call Solr's reload command, a new instance of AnalyzingInfixSuggester is created. When trying to create a new writer on the same Directory a lock cannot be obtained and Solr fails to reload the core. SolrSuggester adds close hooks but that doesn't work in this case. The reason is that the close hooks gets called only after the new core gets created. CoreContainer.reload() first creates the new core ( which fails ) and only after it has registered the new core does it clean up the old core ( registerCore() ) We need to clean up the resources before the new core is created or use the same resources again if it hasn't changed.
          Hide
          varunthacker Varun Thacker added a comment -

          I experimented by adding a closeHook which gets called before the new SolrCore gets created. At least the core now reloads correctly.

          This approach has one major disadvantage - There will be a short period where the suggester won't work since we close it before the new core has been registered.

          Any ideas as to how should we tackle this in a correct way?

          Show
          varunthacker Varun Thacker added a comment - I experimented by adding a closeHook which gets called before the new SolrCore gets created. At least the core now reloads correctly. This approach has one major disadvantage - There will be a short period where the suggester won't work since we close it before the new core has been registered. Any ideas as to how should we tackle this in a correct way?
          Hide
          varunthacker Varun Thacker added a comment -

          Simple test which fails because the core fails to load.

          Show
          varunthacker Varun Thacker added a comment - Simple test which fails because the core fails to load.
          Hide
          varunthacker Varun Thacker added a comment -

          Previous patch was incorrect. Forgot to add the solrconfig changes without which AnalyzingInfixSuggester would never get used in the test.

          This is also reported by an user on the user list - http://lucene.472066.n3.nabble.com/BlendedInfixSuggester-index-write-lock-failures-on-core-reload-td4152984.html

          Show
          varunthacker Varun Thacker added a comment - Previous patch was incorrect. Forgot to add the solrconfig changes without which AnalyzingInfixSuggester would never get used in the test. This is also reported by an user on the user list - http://lucene.472066.n3.nabble.com/BlendedInfixSuggester-index-write-lock-failures-on-core-reload-td4152984.html
          Hide
          hossman Hoss Man added a comment -

          SolrSuggester adds close hooks but that doesn't work in this case. The reason is that the close hooks gets called only after the new core gets created....

          I experimented by adding a closeHook which gets called before the new SolrCore gets created. At least the core now reloads correctly.

          Hmm, yeah ... the lifecycle implications and general API changes involved in fixing this bug are definitely tricky ... i'm not eager to rush into adding a new method to CloseHook. If we do add something, i think it might be better to consider a general "ReloadHook" that could inform components when a SolrCore is about to be reloaded, and then followup with the new SolrCore instance once it's created, maybe something like...

          public abstract ReloadHook {
            public abstract void preReload(SolrCore oldCore);
            public abstract void postReload(SolrCore oldCore, SolrCore newCore);
          }
          

          This approach has one major disadvantage - There will be a short period where the suggester won't work since we close it before the new core has been registered.

          One way we might be able to mitigate that is by: a) changing the lock factory we use on the suggester Directory; 2) subclassing AnalyzingInfixSuggester to be aware of the reloading taking place.

          The idea being that when AnalyzingInfixLookupFactory initially constructs the FSDirectory, it could explicitly configure something like the SingleInstanceLockFactory - that should allow 2 instances of AnalyzingInfixSuggester (in the same JVM) open it at the same time – but then, to prevent corruption risk if both Suggester instances try to write to that Directory, we need to subclass them and customize them to know when the "reload" is taking place, so the old one blocks itself from doing anymore writes.

          so suggestions would still be available while waiting for the new core to start, but not updates to the dictionary.


          This is definitely hairy.

          Show
          hossman Hoss Man added a comment - SolrSuggester adds close hooks but that doesn't work in this case. The reason is that the close hooks gets called only after the new core gets created.... I experimented by adding a closeHook which gets called before the new SolrCore gets created. At least the core now reloads correctly. Hmm, yeah ... the lifecycle implications and general API changes involved in fixing this bug are definitely tricky ... i'm not eager to rush into adding a new method to CloseHook. If we do add something, i think it might be better to consider a general "ReloadHook" that could inform components when a SolrCore is about to be reloaded, and then followup with the new SolrCore instance once it's created, maybe something like... public abstract ReloadHook { public abstract void preReload(SolrCore oldCore); public abstract void postReload(SolrCore oldCore, SolrCore newCore); } This approach has one major disadvantage - There will be a short period where the suggester won't work since we close it before the new core has been registered. One way we might be able to mitigate that is by: a) changing the lock factory we use on the suggester Directory; 2) subclassing AnalyzingInfixSuggester to be aware of the reloading taking place. The idea being that when AnalyzingInfixLookupFactory initially constructs the FSDirectory, it could explicitly configure something like the SingleInstanceLockFactory - that should allow 2 instances of AnalyzingInfixSuggester (in the same JVM) open it at the same time – but then, to prevent corruption risk if both Suggester instances try to write to that Directory, we need to subclass them and customize them to know when the "reload" is taking place, so the old one blocks itself from doing anymore writes. so suggestions would still be available while waiting for the new core to start, but not updates to the dictionary. This is definitely hairy.
          Hide
          janhoy Jan Høydahl added a comment -

          Hmm, this is annoying, what is the next step?

          Show
          janhoy Jan Høydahl added a comment - Hmm, this is annoying, what is the next step?
          Hide
          janhoy Jan Høydahl added a comment -

          Could we perhaps do this in steps?

          1. First implement ReloadHook and accept that suggester is unavailable during reload
          2. Then handle fancy locking stuff in another issue?
          Show
          janhoy Jan Høydahl added a comment - Could we perhaps do this in steps? First implement ReloadHook and accept that suggester is unavailable during reload Then handle fancy locking stuff in another issue?
          Hide
          varunthacker Varun Thacker added a comment - - edited

          I was testing out the AnalyzingInfixSuggester on trunk and noticed that with <str name="buildOnStartup">false</str> ( which is the default ) this problem does not happen. Only when we turned to true does a reload fail.

          So starting Solr 5.1 setting <str name="buildOnStartup">false</str> should be the workaround till we fix this problem

          EDIT: I was wrong. The problem still exists. Sorry for the noise.

          Show
          varunthacker Varun Thacker added a comment - - edited I was testing out the AnalyzingInfixSuggester on trunk and noticed that with <str name="buildOnStartup">false</str> ( which is the default ) this problem does not happen. Only when we turned to true does a reload fail. So starting Solr 5.1 setting <str name="buildOnStartup">false</str> should be the workaround till we fix this problem EDIT: I was wrong. The problem still exists. Sorry for the noise.
          Hide
          stephlag Stephan Lagraulet added a comment -

          Hoss Man How would you add this ReloadHook? I was thinking of adding methods to SolrCoreAware but there's a significant impact as many classes uses this interface...

          Show
          stephlag Stephan Lagraulet added a comment - Hoss Man How would you add this ReloadHook? I was thinking of adding methods to SolrCoreAware but there's a significant impact as many classes uses this interface...
          Hide
          hossman Hoss Man added a comment -

          the crux of my suggestion was that we should NOT add any new methods to any existing interface/abstraction, instead we could add an entirely new ReloadHook API, and let classes pass instances of that API to Solr just like they can currently pass instances of CloseHook.

          it was a largely off the cuff suggestion, that i haven't really given any thought to since ... i have no idea if there is a better approach.

          Jan: if you think it's a good idea, then by all means go for it piecemeal ... i honestly have no idea how serious the downsides would be during the interim between adding a ReloadHook that the Suggester could use in one version of solr and resolving the locking issues in a later version (corrupt suggestions index after reload? errors when trying to add docs during reload? complete failure to reload?) ... those ramifications seem like they should make the difference in deciding wether the changes can be made incrementally.

          Show
          hossman Hoss Man added a comment - the crux of my suggestion was that we should NOT add any new methods to any existing interface/abstraction, instead we could add an entirely new ReloadHook API, and let classes pass instances of that API to Solr just like they can currently pass instances of CloseHook. it was a largely off the cuff suggestion, that i haven't really given any thought to since ... i have no idea if there is a better approach. Jan: if you think it's a good idea, then by all means go for it piecemeal ... i honestly have no idea how serious the downsides would be during the interim between adding a ReloadHook that the Suggester could use in one version of solr and resolving the locking issues in a later version (corrupt suggestions index after reload? errors when trying to add docs during reload? complete failure to reload?) ... those ramifications seem like they should make the difference in deciding wether the changes can be made incrementally.
          Hide
          dsmiley David Smiley added a comment -

          I admit I haven't looked at the patch or underlying code, but perhaps no new API is needed to solve this. Perhaps suggesters that write to Lucene directories could (somehow) be configured to only use a lock file when it needs to build the index? If a suggester-build is in progress on the old core, this might fail a reload attempt... but this is a far smaller issue than the disaster we have today.

          Show
          dsmiley David Smiley added a comment - I admit I haven't looked at the patch or underlying code, but perhaps no new API is needed to solve this. Perhaps suggesters that write to Lucene directories could (somehow) be configured to only use a lock file when it needs to build the index? If a suggester-build is in progress on the old core, this might fail a reload attempt... but this is a far smaller issue than the disaster we have today.
          Hide
          alessandro.benedetti Alessandro Benedetti added a comment -

          Any update on this ?
          is Anyone working on this at the minute ?
          I agree is quite important as basically is not possible to consistently use the suggester when Solr in Cloud mode !

          Show
          alessandro.benedetti Alessandro Benedetti added a comment - Any update on this ? is Anyone working on this at the minute ? I agree is quite important as basically is not possible to consistently use the suggester when Solr in Cloud mode !
          Hide
          jacquesdr@gmail.com Jacques Du Rand added a comment -

          Confirmed. Bug still exists in Solr 5.3.1

          Show
          jacquesdr@gmail.com Jacques Du Rand added a comment - Confirmed. Bug still exists in Solr 5.3.1
          Hide
          dpgover Dario Govergun added a comment - - edited

          I can also confirm that the bug still exists in Solr 5.3.1
          And I also tested it in 5.5.0 with the same luck. It locks.

          Show
          dpgover Dario Govergun added a comment - - edited I can also confirm that the bug still exists in Solr 5.3.1 And I also tested it in 5.5.0 with the same luck. It locks.
          Hide
          janhoy Jan Høydahl added a comment -

          I was thinking more that in preReload() the old Suggester simply shuts down, freeing up any locks and making suggest unavailable. Then the reload will not fail due to locks, but there is a (potentially long) time span where suggest is unavailable. I feel this is better than failing reloads, which in most environments do not take place very frequently after all.

          Show
          janhoy Jan Høydahl added a comment - I was thinking more that in preReload() the old Suggester simply shuts down, freeing up any locks and making suggest unavailable. Then the reload will not fail due to locks, but there is a (potentially long) time span where suggest is unavailable. I feel this is better than failing reloads, which in most environments do not take place very frequently after all.
          Hide
          dsmiley David Smiley added a comment -

          +1 to this compromise for the time being

          Show
          dsmiley David Smiley added a comment - +1 to this compromise for the time being
          Hide
          dpgover Dario Govergun added a comment -

          I don't know if the other issue I'm having while implementing this has something to do with this one. If not, please correct me.

          The other thing that is happening to me is that I get a Store Lookup Build Failed error, after sending a build command, using the suggester configured to use the AnalyzingInfixSuggester implementation.
          The logs are not very explicit:

          2016-03-17 18:07:02.616 INFO  (qtp1698904557-13) [   x:users] o.a.s.h.c.SuggestComponent SuggestComponent prepare with : omitHeader=true&echoParams=explicit&suggest.dictionary=suggest&suggest.build=true&suggest=true&json.nl=flat&wt=json&suggest.count=10
          2016-03-17 18:07:02.618 INFO  (qtp1698904557-13) [   x:users] o.a.s.s.s.SolrSuggester SolrSuggester.build(suggest)
          2016-03-17 18:07:02.778 ERROR (qtp1698904557-13) [   x:users] o.a.s.s.s.SolrSuggester Store Lookup build failed
          2016-03-17 18:07:02.778 INFO  (qtp1698904557-13) [   x:users] o.a.s.h.c.SuggestComponent SuggestComponent process with : omitHeader=true&echoParams=explicit&suggest.dictionary=suggest&suggest.build=true&suggest=true&json.nl=flat&wt=json&suggest.count=10
          2016-03-17 18:07:02.778 INFO  (qtp1698904557-13) [   x:users] o.a.s.c.S.Request [users] webapp=/solr path=/suggest params={omitHeader=true&suggest=true&suggest.build=true&json.nl=flat&wt=json} status=0 QTime=162
          

          Does this have to do with the same thing? Or is it another issue with this lookup implementation?

          Show
          dpgover Dario Govergun added a comment - I don't know if the other issue I'm having while implementing this has something to do with this one. If not, please correct me. The other thing that is happening to me is that I get a Store Lookup Build Failed error, after sending a build command, using the suggester configured to use the AnalyzingInfixSuggester implementation. The logs are not very explicit: 2016-03-17 18:07:02.616 INFO (qtp1698904557-13) [ x:users] o.a.s.h.c.SuggestComponent SuggestComponent prepare with : omitHeader= true &echoParams=explicit&suggest.dictionary=suggest&suggest.build= true &suggest= true &json.nl=flat&wt=json&suggest.count=10 2016-03-17 18:07:02.618 INFO (qtp1698904557-13) [ x:users] o.a.s.s.s.SolrSuggester SolrSuggester.build(suggest) 2016-03-17 18:07:02.778 ERROR (qtp1698904557-13) [ x:users] o.a.s.s.s.SolrSuggester Store Lookup build failed 2016-03-17 18:07:02.778 INFO (qtp1698904557-13) [ x:users] o.a.s.h.c.SuggestComponent SuggestComponent process with : omitHeader= true &echoParams=explicit&suggest.dictionary=suggest&suggest.build= true &suggest= true &json.nl=flat&wt=json&suggest.count=10 2016-03-17 18:07:02.778 INFO (qtp1698904557-13) [ x:users] o.a.s.c.S.Request [users] webapp=/solr path=/suggest params={omitHeader= true &suggest= true &suggest.build= true &json.nl=flat&wt=json} status=0 QTime=162 Does this have to do with the same thing? Or is it another issue with this lookup implementation?
          Hide
          gquaire Gérald Quaire added a comment -

          Hello,

          I met this issue in my project. I need to reload the Solr Core after modifying the configuration of the suggester via Solrj. In my suggester, I 'm using an AnalyzingInfixSuggester as lookup algorithm. At each reload command, the "LockObtainFailedException" exception rises.
          To avoid this problem, I have overloaded the SuggestComponent and the SolrSuggester classes in order to introduce a static map that stores the suggesters already created for the current core and the current composant. My SolrSuggester is now implemented the Closeable interface to call the close method of the lookup object.
          So when the core is reloading, the SuggestComponent first gets the suggesters created previously by this core and closes all suggesters. And then, it can create the new Suggester instances. Here is an excerpt of the code in the SuggestComponent:

          protected static Map<String, Map<String, SolrSuggester>> CoreSuggesters = new ConcurrentHashMap<>();
          ...
          @Override
          public void inform(SolrCore core) {
          if (initParams != null) {
          LOG.info("Initializing SuggestComponent");

          CoreSuggesters.computeIfPresent(core.getName() + this.getName(), (K, map) -> {
          if (map != null) {
          for (SolrSuggester suggest : map.values()) {
          try

          { suggest.close(); }

          catch (IOException e)

          { LOG.warn("Could not close the suggester.", e); }

          }
          map.clear();
          }
          return null;
          });

          // Initialize the new suggesters here
          ...
          CoreSuggesters.putIfAbsent(core.getName() + this.getName(), suggesters);
          core.addCloseHook(new CloseHook() {
          @Override
          public void preClose(SolrCore core) {
          CoreSuggesters.computeIfPresent(core.getName() + internalName, (K, map) -> {
          if (map != null) {
          for (SolrSuggester suggest : map.values()) {
          try

          { suggest.close(); }

          catch (IOException e)

          { LOG.warn("Could not close the suggester.", e); }

          }
          map.clear();
          }
          return null;
          });
          } // end of the inform method

          It was painful to make the overloadingbecause the classes SuggestComponent and SolrSuggester are not written to be extended.
          This code has fixed my issue for now. I don't know if it is a clean solution (I don't think so), but it seems working. I hope this trick will be helpful.

          Show
          gquaire Gérald Quaire added a comment - Hello, I met this issue in my project. I need to reload the Solr Core after modifying the configuration of the suggester via Solrj. In my suggester, I 'm using an AnalyzingInfixSuggester as lookup algorithm. At each reload command, the "LockObtainFailedException" exception rises. To avoid this problem, I have overloaded the SuggestComponent and the SolrSuggester classes in order to introduce a static map that stores the suggesters already created for the current core and the current composant. My SolrSuggester is now implemented the Closeable interface to call the close method of the lookup object. So when the core is reloading, the SuggestComponent first gets the suggesters created previously by this core and closes all suggesters. And then, it can create the new Suggester instances. Here is an excerpt of the code in the SuggestComponent: protected static Map<String, Map<String, SolrSuggester>> CoreSuggesters = new ConcurrentHashMap<>(); ... @Override public void inform(SolrCore core) { if (initParams != null) { LOG.info("Initializing SuggestComponent"); CoreSuggesters.computeIfPresent(core.getName() + this.getName(), (K, map) -> { if (map != null) { for (SolrSuggester suggest : map.values()) { try { suggest.close(); } catch (IOException e) { LOG.warn("Could not close the suggester.", e); } } map.clear(); } return null; }); // Initialize the new suggesters here ... CoreSuggesters.putIfAbsent(core.getName() + this.getName(), suggesters); core.addCloseHook(new CloseHook() { @Override public void preClose(SolrCore core) { CoreSuggesters.computeIfPresent(core.getName() + internalName, (K, map) -> { if (map != null) { for (SolrSuggester suggest : map.values()) { try { suggest.close(); } catch (IOException e) { LOG.warn("Could not close the suggester.", e); } } map.clear(); } return null; }); } // end of the inform method It was painful to make the overloadingbecause the classes SuggestComponent and SolrSuggester are not written to be extended. This code has fixed my issue for now. I don't know if it is a clean solution (I don't think so), but it seems working. I hope this trick will be helpful.
          Hide
          shamik Shamik Bandopadhyay added a comment -

          Any update on this ?

          Show
          shamik Shamik Bandopadhyay added a comment - Any update on this ?
          Hide
          varunthacker Varun Thacker added a comment -

          Hi Shamik,

          I don't think anyone is currently working on it. But patches are always welcome

          Show
          varunthacker Varun Thacker added a comment - Hi Shamik, I don't think anyone is currently working on it. But patches are always welcome
          Hide
          gsingers Grant Ingersoll added a comment -

          I've been hitting this as well. One suggestion that might minimize the impact: close the writer after build. In analyzing the method usages, we don't seem to call the inline/online "add/update" methods anywhere in the code, so there really isn't any reason to keep the writer open. I know in theory Lucene supports updating the suggesters, but we aren't using it.

          While closing the writer at the end of the build won't completely eliminate the problem (e.g. opening a new searcher while a build is underway), I think it could lower the likelihood.

          Show
          gsingers Grant Ingersoll added a comment - I've been hitting this as well. One suggestion that might minimize the impact: close the writer after build. In analyzing the method usages, we don't seem to call the inline/online "add/update" methods anywhere in the code, so there really isn't any reason to keep the writer open. I know in theory Lucene supports updating the suggesters, but we aren't using it. While closing the writer at the end of the build won't completely eliminate the problem (e.g. opening a new searcher while a build is underway), I think it could lower the likelihood.
          Hide
          varunthacker Varun Thacker added a comment -

          close the writer after build. In analyzing the method usages

          I think that might be a good solution.

          Solr does not support real time updates to the suggester so its fine from it's perspective.

          From a Lucene API standpoint it's probably not ideal but maybe not all that bad?

          This is what I am thinking -

          Create a Lucene issue in which AnalyzingInfixSuggester#build closes the writer by default at the end.

          The add and update methods call ensureOpen and those who do frequent real time updates directly via lucene won't see any slowdowns.

          Michael McCandless - Would this approach have any major drawback from Lucene's perspective? Else I can go ahead an tackle this in a Lucene issue

          Show
          varunthacker Varun Thacker added a comment - close the writer after build. In analyzing the method usages I think that might be a good solution. Solr does not support real time updates to the suggester so its fine from it's perspective. From a Lucene API standpoint it's probably not ideal but maybe not all that bad? This is what I am thinking - Create a Lucene issue in which AnalyzingInfixSuggester#build closes the writer by default at the end. The add and update methods call ensureOpen and those who do frequent real time updates directly via lucene won't see any slowdowns. Michael McCandless - Would this approach have any major drawback from Lucene's perspective? Else I can go ahead an tackle this in a Lucene issue
          Hide
          mikemccand Michael McCandless added a comment -

          Fixing AnalyzingInfixSuggester to close the writer at the end of build seems reasonable?

          Show
          mikemccand Michael McCandless added a comment - Fixing AnalyzingInfixSuggester to close the writer at the end of build seems reasonable?
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user Peter-LaComb opened a pull request:

          https://github.com/apache/lucene-solr/pull/96

          SOLR-6246 - Fix core reload if suggester has been built.

          In my testing, it is not required to keep the writer open for the suggester to keep working.
          Add and Update call ensureOpen, which will open a new writer if it has been set = null.
          This change closes it at the end of a build and sets the reference = null such that
          Add and Update will continue to work correctly. Additionally, commit is updated to not
          throw if the writer is null. This is correct because nothing has been added or updated
          since the last build.
          The only thing I'm left with uncertainty about is reloading a core with NRT updates
          pending. This would appear to still cause the issue to appear again. The difference being that
          a rebuild would alleviate the issue. This requires additional thought.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/Peter-LaComb/lucene-solr bugfix/SOLR-6246

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/96.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #96


          commit c31db2f53b431ccb3263c824c7d5fde20aab5293
          Author: Peter T. LaComb Jr <peter.lacomb@beeline.com>
          Date: 2016-10-13T20:28:25Z

          SOLR-6246 - Fix core reload if suggester has been built.
          In my testing, it is not required to keep the writer open for the suggester to keep working.
          Add and Update call ensureOpen, which will open a new writer if it has been set = null.
          This change closes it at the end of a build and sets the reference = null such that
          Add and Update will continue to work correctly. Additionally, commit is updated to not
          throw if the writer is null. This is correct because nothing has been added or updated
          since the last build.
          The only thing I'm left with uncertainty about is reloading a core with NRT updates
          pending. This would appear to still cause the issue to appear again. The difference being that
          a rebuild would alleviate the issue. This requires additional thought.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user Peter-LaComb opened a pull request: https://github.com/apache/lucene-solr/pull/96 SOLR-6246 - Fix core reload if suggester has been built. In my testing, it is not required to keep the writer open for the suggester to keep working. Add and Update call ensureOpen, which will open a new writer if it has been set = null. This change closes it at the end of a build and sets the reference = null such that Add and Update will continue to work correctly. Additionally, commit is updated to not throw if the writer is null. This is correct because nothing has been added or updated since the last build. The only thing I'm left with uncertainty about is reloading a core with NRT updates pending. This would appear to still cause the issue to appear again. The difference being that a rebuild would alleviate the issue. This requires additional thought. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Peter-LaComb/lucene-solr bugfix/ SOLR-6246 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/96.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #96 commit c31db2f53b431ccb3263c824c7d5fde20aab5293 Author: Peter T. LaComb Jr <peter.lacomb@beeline.com> Date: 2016-10-13T20:28:25Z SOLR-6246 - Fix core reload if suggester has been built. In my testing, it is not required to keep the writer open for the suggester to keep working. Add and Update call ensureOpen, which will open a new writer if it has been set = null. This change closes it at the end of a build and sets the reference = null such that Add and Update will continue to work correctly. Additionally, commit is updated to not throw if the writer is null. This is correct because nothing has been added or updated since the last build. The only thing I'm left with uncertainty about is reloading a core with NRT updates pending. This would appear to still cause the issue to appear again. The difference being that a rebuild would alleviate the issue. This requires additional thought.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Peter-LaComb commented on the issue:

          https://github.com/apache/lucene-solr/pull/96

          I broke three or more tests - need to fix that.

          Show
          githubbot ASF GitHub Bot added a comment - Github user Peter-LaComb commented on the issue: https://github.com/apache/lucene-solr/pull/96 I broke three or more tests - need to fix that.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Peter-LaComb closed the pull request at:

          https://github.com/apache/lucene-solr/pull/96

          Show
          githubbot ASF GitHub Bot added a comment - Github user Peter-LaComb closed the pull request at: https://github.com/apache/lucene-solr/pull/96
          Hide
          steve_rowe Steve Rowe added a comment -

          One suggestion that might minimize the impact: close the writer after build.

          I've opened LUCENE-7564 to do this.

          Show
          steve_rowe Steve Rowe added a comment - One suggestion that might minimize the impact: close the writer after build. I've opened LUCENE-7564 to do this.
          Hide
          steve_rowe Steve Rowe added a comment - - edited

          Modernized version of Varun Thacker's test patch.

          This test now succeeds when I run it on master patched with LUCENE-7564.

          There is no testing of reloading while build is underway though.

          Show
          steve_rowe Steve Rowe added a comment - - edited Modernized version of Varun Thacker 's test patch. This test now succeeds when I run it on master patched with LUCENE-7564 . There is no testing of reloading while build is underway though.
          Hide
          Christoph Froeschel Christoph Froeschel added a comment -

          Hello.
          I wanted to hear if there is an update to this issue or a workaround?
          We are seeing this issue in SOLR version 6.2.1.

          Show
          Christoph Froeschel Christoph Froeschel added a comment - Hello. I wanted to hear if there is an update to this issue or a workaround? We are seeing this issue in SOLR version 6.2.1.
          Hide
          t-rude Andreas Ravn added a comment -

          I did a test installation of last week's nightly build of the forthcoming 6.4, and could not reproduce the issue anymore. I was able to flawlessly reload indexes that were using AnalyzingInfixSuggester. As far as a couple of functionality tests were going, everything seems to work fine.

          Nevertheless, I would also be interested in obtaining more "official" information on the fix's status.

          Show
          t-rude Andreas Ravn added a comment - I did a test installation of last week's nightly build of the forthcoming 6.4, and could not reproduce the issue anymore. I was able to flawlessly reload indexes that were using AnalyzingInfixSuggester. As far as a couple of functionality tests were going, everything seems to work fine. Nevertheless, I would also be interested in obtaining more "official" information on the fix's status.
          Hide
          erickerickson Erick Erickson added a comment -

          Perhaps related?

          Show
          erickerickson Erick Erickson added a comment - Perhaps related?
          Hide
          dsmiley David Smiley added a comment -

          Perhaps related?

          Erick Erickson what is related to what?

          Show
          dsmiley David Smiley added a comment - Perhaps related? Erick Erickson what is related to what?
          Hide
          erickerickson Erick Erickson added a comment -

          Bah. The comments when linking other JIRAs don't have any obvious association, do they?

          Perhaps related to SOLR-7747.

          Show
          erickerickson Erick Erickson added a comment - Bah. The comments when linking other JIRAs don't have any obvious association, do they? Perhaps related to SOLR-7747 .
          Hide
          Christoph Froeschel Christoph Froeschel added a comment -

          Just wanted to hear if there is someone having a workaround for this?

          Best Regards
          Christoph

          Show
          Christoph Froeschel Christoph Froeschel added a comment - Just wanted to hear if there is someone having a workaround for this? Best Regards Christoph
          Hide
          erickerickson Erick Erickson added a comment -

          It'd be great if you could give a recent Solr a spin and see if the problem persists.

          There are a bunch of JIRAs floating around with some hints that this is fixed but nobody has volunteered to verify that this JIRA is fixed by them.
          See:
          LUCENE-7564
          SOLR-6100

          Show
          erickerickson Erick Erickson added a comment - It'd be great if you could give a recent Solr a spin and see if the problem persists. There are a bunch of JIRAs floating around with some hints that this is fixed but nobody has volunteered to verify that this JIRA is fixed by them. See: LUCENE-7564 SOLR-6100
          Hide
          t-rude Andreas Ravn added a comment -

          Hi Erick,

          I did. Please see my reply to Christoph Froeschels comment from Dec 19, 2016.

          Cheers, Andreas

          Show
          t-rude Andreas Ravn added a comment - Hi Erick, I did. Please see my reply to Christoph Froeschels comment from Dec 19, 2016. Cheers, Andreas
          Hide
          jdoepp James Doepp added a comment -

          My "workaround" was to create a separate core with only the suggesters.

          Show
          jdoepp James Doepp added a comment - My "workaround" was to create a separate core with only the suggesters.
          Hide
          erickerickson Erick Erickson added a comment -

          Thanks Andreas!

          Show
          erickerickson Erick Erickson added a comment - Thanks Andreas!
          Hide
          Christoph Froeschel Christoph Froeschel added a comment - - edited

          Hello.

          I've looked at Andreas' comment. The problem is that i need to push some software to production and for that I would not like to have a nightly build but a stable release. On my test machine I have 6.2.1 installed. That's why I was looking for a workaround.

          Best Regards
          Christoph

          Show
          Christoph Froeschel Christoph Froeschel added a comment - - edited Hello. I've looked at Andreas' comment. The problem is that i need to push some software to production and for that I would not like to have a nightly build but a stable release. On my test machine I have 6.2.1 installed. That's why I was looking for a workaround. Best Regards Christoph
          Hide
          erickerickson Erick Erickson added a comment -

          If you can stand to wait a bit, Solr 6.4 may be cut in the next couple of weeks. Otherwise the only workaround I've seen is to have a separate core be your suggester that you don't reload.

          Show
          erickerickson Erick Erickson added a comment - If you can stand to wait a bit, Solr 6.4 may be cut in the next couple of weeks. Otherwise the only workaround I've seen is to have a separate core be your suggester that you don't reload.
          Hide
          yuanyun.cn jefferyyuan added a comment -

          I am running on Solr 6.4 - solr-6.4.0-195.
          But the problem still exists. Even restarting solr doesn't work - after restart solr and reload collection or current node still fails with LockObtainFailedException.

          I even tried to manually delete the write.lock, then call reload-collection/cores , it still failed again with same error.

          INFO - 2017-01-12 16:55:42.392; [c:myCollection s:shard2 r:core_node3 x:searchItems_shard2_replica1] org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null path=/admin/cores params=

          {core=searchItems_shard2_replica1&qt=/admin/cores&action=RELOAD&wt=javabin&version=2}

          status=500 QTime=592
          ERROR - 2017-01-12 16:55:42.393; [c:myCollection s:shard2 r:core_node3 x:searchItems_shard2_replica1] org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: Error handling 'reload' action
          at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$2(CoreAdminOperation.java:114)
          at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$23/265321659.execute(Unknown Source)
          at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:377)
          at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:365)
          at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:156)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:152)
          at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
          at org.eclipse.jetty.server.Server.handle(Server.java:534)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
          at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
          at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
          at java.lang.Thread.run(Thread.java:745)
          Caused by: org.apache.solr.common.SolrException: Unable to reload core [searchItems_shard2_replica1]
          at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950)
          at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$2(CoreAdminOperation.java:112)
          ... 34 more
          Caused by: org.apache.solr.common.SolrException: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Applications/solr-6.4.0/example/cloud/node2/solr/searchItems_shard2_replica1/data/infix_suggestions/write.lock
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:899)
          at org.apache.solr.core.SolrCore.reload(SolrCore.java:589)
          at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:944)
          ... 35 more
          Caused by: java.lang.RuntimeException: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Applications/solr-6.4.0/example/cloud/node2/solr/searchItems_shard2_replica1/data/infix_suggestions/write.lock
          at org.apache.solr.spelling.suggest.fst.BlendedInfixLookupFactory.create(BlendedInfixLookupFactory.java:139)
          at org.apache.solr.spelling.suggest.SolrSuggester.init(SolrSuggester.java:120)
          at org.apache.solr.handler.component.SuggestComponent.inform(SuggestComponent.java:119)
          at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:695)
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:879)
          ... 37 more
          Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Applications/solr-6.4.0/example/cloud/node2/solr/searchItems_shard2_replica1/data/infix_suggestions/write.lock
          at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127)
          at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
          at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
          at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804)
          at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.<init>(AnalyzingInfixSuggester.java:250)
          at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.<init>(AnalyzingInfixSuggester.java:207)
          at org.apache.lucene.search.suggest.analyzing.BlendedInfixSuggester.<init>(BlendedInfixSuggester.java:141)
          at org.apache.solr.spelling.suggest.fst.BlendedInfixLookupFactory$1.<init>(BlendedInfixLookupFactory.java:119)
          at org.apache.solr.spelling.suggest.fst.BlendedInfixLookupFactory.create(BlendedInfixLookupFactory.java:116)
          ... 41 more

          Show
          yuanyun.cn jefferyyuan added a comment - I am running on Solr 6.4 - solr-6.4.0-195. But the problem still exists. Even restarting solr doesn't work - after restart solr and reload collection or current node still fails with LockObtainFailedException. I even tried to manually delete the write.lock, then call reload-collection/cores , it still failed again with same error. INFO - 2017-01-12 16:55:42.392; [c:myCollection s:shard2 r:core_node3 x:searchItems_shard2_replica1] org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null path=/admin/cores params= {core=searchItems_shard2_replica1&qt=/admin/cores&action=RELOAD&wt=javabin&version=2} status=500 QTime=592 ERROR - 2017-01-12 16:55:42.393; [c:myCollection s:shard2 r:core_node3 x:searchItems_shard2_replica1] org.apache.solr.common.SolrException; null:org.apache.solr.common.SolrException: Error handling 'reload' action at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$2(CoreAdminOperation.java:114) at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$23/265321659.execute(Unknown Source) at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:377) at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:365) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:156) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:152) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Unable to reload core [searchItems_shard2_replica1] at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:950) at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$2(CoreAdminOperation.java:112) ... 34 more Caused by: org.apache.solr.common.SolrException: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Applications/solr-6.4.0/example/cloud/node2/solr/searchItems_shard2_replica1/data/infix_suggestions/write.lock at org.apache.solr.core.SolrCore.<init>(SolrCore.java:899) at org.apache.solr.core.SolrCore.reload(SolrCore.java:589) at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:944) ... 35 more Caused by: java.lang.RuntimeException: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Applications/solr-6.4.0/example/cloud/node2/solr/searchItems_shard2_replica1/data/infix_suggestions/write.lock at org.apache.solr.spelling.suggest.fst.BlendedInfixLookupFactory.create(BlendedInfixLookupFactory.java:139) at org.apache.solr.spelling.suggest.SolrSuggester.init(SolrSuggester.java:120) at org.apache.solr.handler.component.SuggestComponent.inform(SuggestComponent.java:119) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:695) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:879) ... 37 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Applications/solr-6.4.0/example/cloud/node2/solr/searchItems_shard2_replica1/data/infix_suggestions/write.lock at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127) at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804) at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.<init>(AnalyzingInfixSuggester.java:250) at org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester.<init>(AnalyzingInfixSuggester.java:207) at org.apache.lucene.search.suggest.analyzing.BlendedInfixSuggester.<init>(BlendedInfixSuggester.java:141) at org.apache.solr.spelling.suggest.fst.BlendedInfixLookupFactory$1.<init>(BlendedInfixLookupFactory.java:119) at org.apache.solr.spelling.suggest.fst.BlendedInfixLookupFactory.create(BlendedInfixLookupFactory.java:116) ... 41 more
          Hide
          steve_rowe Steve Rowe added a comment -

          jefferyyuan, Solr 6.4 has not yet been released - where did "solr-6.4.0-195" come from?

          Show
          steve_rowe Steve Rowe added a comment - jefferyyuan , Solr 6.4 has not yet been released - where did "solr-6.4.0-195" come from?
          Hide
          yuanyun.cn jefferyyuan added a comment -

          From https://builds.apache.org/job/Solr-Artifacts-6.x/lastSuccessfulBuild/artifact/solr/package/

          • it was build 195 when I downloaded at that time

          I will try the newest build and check whether this works

          Show
          yuanyun.cn jefferyyuan added a comment - From https://builds.apache.org/job/Solr-Artifacts-6.x/lastSuccessfulBuild/artifact/solr/package/ it was build 195 when I downloaded at that time I will try the newest build and check whether this works
          Hide
          yuanyun.cn jefferyyuan added a comment - - edited

          I tested with the latest build - solr-6.4.0-222,reloading collection/cores with AnalyzingInfixSuggester still failed with LockObtainFailedException.
          It failed with same error even after after solr.

          It can be easily reproduced, add a suggest component, then **build the suggester**: suggest?suggest.build=true. Then reload collection or cores.

          • Seems the key to reproduce the issue is we need build the suggester.

          <searchComponent name="suggest" class="solr.SuggestComponent">
          <lst name="suggester">
          <str name="name">infixSuggester</str>
          <str name="lookupImpl">BlendedInfixLookupFactory</str>
          <str name="dictionaryImpl">DocumentDictionaryFactory</str>
          <str name="blenderType">position_linear</str>
          <str name="field">suggester</str>
          <str name="contextField">suggesterContextField</str>
          <str name="minPrefixChars">4</str>
          <str name="suggestAnalyzerFieldType">textSuggest</str>
          <str name="indexPath">infix_suggestions</str>
          <str name="highlight">true</str>
          <str name="buildOnStartup">false</str>
          <str name="buildOnCommit">false</str>
          </lst>
          </searchComponent>

          <requestHandler name="/suggest" class="solr.SearchHandler"
          >
          <lst name="defaults">
          <str name="suggest">true</str>
          <str name="suggest.dictionary">infixSuggester</str>
          <str name="suggest.onlyMorePopular">true</str>
          <str name="suggest.count">10</str>
          <str name="suggest.collate">true</str>
          </lst>
          <arr name="components">
          <str>suggest</str>
          </arr>
          </requestHandler>

          Show
          yuanyun.cn jefferyyuan added a comment - - edited I tested with the latest build - solr-6.4.0-222,reloading collection/cores with AnalyzingInfixSuggester still failed with LockObtainFailedException. It failed with same error even after after solr. It can be easily reproduced, add a suggest component, then ** build the suggester **: suggest?suggest.build=true. Then reload collection or cores. Seems the key to reproduce the issue is we need build the suggester. <searchComponent name="suggest" class="solr.SuggestComponent"> <lst name="suggester"> <str name="name">infixSuggester</str> <str name="lookupImpl">BlendedInfixLookupFactory</str> <str name="dictionaryImpl">DocumentDictionaryFactory</str> <str name="blenderType">position_linear</str> <str name="field">suggester</str> <str name="contextField">suggesterContextField</str> <str name="minPrefixChars">4</str> <str name="suggestAnalyzerFieldType">textSuggest</str> <str name="indexPath">infix_suggestions</str> <str name="highlight">true</str> <str name="buildOnStartup">false</str> <str name="buildOnCommit">false</str> </lst> </searchComponent> <requestHandler name="/suggest" class="solr.SearchHandler" > <lst name="defaults"> <str name="suggest">true</str> <str name="suggest.dictionary">infixSuggester</str> <str name="suggest.onlyMorePopular">true</str> <str name="suggest.count">10</str> <str name="suggest.collate">true</str> </lst> <arr name="components"> <str>suggest</str> </arr> </requestHandler>
          Hide
          steve_rowe Steve Rowe added a comment -

          Patch that adds a random dictionary with configurable size and new tests that use it:

          1. A build/reload test for each of AnalyzingInfixSuggester and BlendedInfixSuggester.
          2. A test that starts building a suggester in the background then initiates a core reload.

          All the tests fail now. The first two only fail 50% of the time, with the cause given in others' reports on this issue; I'm not sure why they don't fail all the time. (My earlier report of the previous test patch passing may have been luck/insufficient trials?). The reload-while-building test sometimes finishes building, and then fails like the other tests, but sometimes reload causes the suggester's index writer to be closed, which causes an exception to be thrown, interrupting the build process.

          For some reason, the suggesters' sidecar indexes have write.lock files in them even after writer.close() is called.

          Show
          steve_rowe Steve Rowe added a comment - Patch that adds a random dictionary with configurable size and new tests that use it: A build/reload test for each of AnalyzingInfixSuggester and BlendedInfixSuggester . A test that starts building a suggester in the background then initiates a core reload. All the tests fail now. The first two only fail 50% of the time, with the cause given in others' reports on this issue; I'm not sure why they don't fail all the time. (My earlier report of the previous test patch passing may have been luck/insufficient trials?). The reload-while-building test sometimes finishes building, and then fails like the other tests, but sometimes reload causes the suggester's index writer to be closed, which causes an exception to be thrown, interrupting the build process. For some reason, the suggesters' sidecar indexes have write.lock files in them even after writer.close() is called.
          Hide
          steve_rowe Steve Rowe added a comment -

          I filed LUCENE-7670 to address an issue I see with tests: a second core reload after a suggester build will trigger the failures mentioned above, because suggesters opened over already-built indexes were creating an IndexWriter just to create a SearcherManager, thus locking the index directory.

          Show
          steve_rowe Steve Rowe added a comment - I filed LUCENE-7670 to address an issue I see with tests: a second core reload after a suggester build will trigger the failures mentioned above, because suggesters opened over already-built indexes were creating an IndexWriter just to create a SearcherManager, thus locking the index directory.
          Hide
          steve_rowe Steve Rowe added a comment -

          jefferyyuan, can you see if the LUCENE-7670 change included in the 6.4.1 release candidate (and also the 6.5.0 snapshot available from https://builds.apache.org/job/Solr-Artifacts-6.x/lastSuccessfulBuild/artifact/solr/package/) fixes the problem for you? The 6.4.1 release candidate #1 is pointed to from here: <https://lists.apache.org/thread.html/89506bf377f49835cfb3d2174d747a5995bfbb5dd9c0475e6ce70aae@%3Cdev.lucene.apache.org%3E>

          Show
          steve_rowe Steve Rowe added a comment - jefferyyuan , can you see if the LUCENE-7670 change included in the 6.4.1 release candidate (and also the 6.5.0 snapshot available from https://builds.apache.org/job/Solr-Artifacts-6.x/lastSuccessfulBuild/artifact/solr/package/ ) fixes the problem for you? The 6.4.1 release candidate #1 is pointed to from here: < https://lists.apache.org/thread.html/89506bf377f49835cfb3d2174d747a5995bfbb5dd9c0475e6ce70aae@%3Cdev.lucene.apache.org%3E >
          Hide
          yuanyun.cn jefferyyuan added a comment -

          First I reproduce the issue in current 6.4.
          Then verified the 6.4.1 release candidate fixed the issue.

          Thanks for solving this issue and looking for 6.4.1 release. Steve Rowe

          Show
          yuanyun.cn jefferyyuan added a comment - First I reproduce the issue in current 6.4. Then verified the 6.4.1 release candidate fixed the issue. Thanks for solving this issue and looking for 6.4.1 release. Steve Rowe
          Hide
          steve_rowe Steve Rowe added a comment -

          I think suggester builds interrupted by core reload or shutdown should fail gracefully (rather than hold up reload/shutdown).

          This patch:

          1. Cleans up the previously described tests.
          2. Changes SolrSuggester.build() to throw SolrCoreState.CoreIsClosedException with a descriptive message when AlreadyClosedException is thrown during suggester build due to a closed IndexWriter (the result of calling close() on the suggester as part of a core reload or shutdown).
          3. Tests for reload & shutdown during suggester build look for the appropriate wrapped exception.
          4. New expectThrows() variant added to LuceneTestCase that checks for both expected outer and wrapped exceptions.

          I think it's ready.

          Show
          steve_rowe Steve Rowe added a comment - I think suggester builds interrupted by core reload or shutdown should fail gracefully (rather than hold up reload/shutdown). This patch: Cleans up the previously described tests. Changes SolrSuggester.build() to throw SolrCoreState.CoreIsClosedException with a descriptive message when AlreadyClosedException is thrown during suggester build due to a closed IndexWriter (the result of calling close() on the suggester as part of a core reload or shutdown). Tests for reload & shutdown during suggester build look for the appropriate wrapped exception. New expectThrows() variant added to LuceneTestCase that checks for both expected outer and wrapped exceptions. I think it's ready.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7b081e468a97b353a5e096ed69163ee9c3044925 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7b081e4 ]

          SOLR-6246: SolrSuggester.build() now throws SolrCoreState.CoreIsClosedException when a core reload/shutdown happens; add a random test lookup dictionary with configurable size; add

          {Analyzing,Blended}

          InfixSuggester reload/build tests; add a wrapped-exception expectThrows() variant to LuceneTestCase

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7b081e468a97b353a5e096ed69163ee9c3044925 in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7b081e4 ] SOLR-6246 : SolrSuggester.build() now throws SolrCoreState.CoreIsClosedException when a core reload/shutdown happens; add a random test lookup dictionary with configurable size; add {Analyzing,Blended} InfixSuggester reload/build tests; add a wrapped-exception expectThrows() variant to LuceneTestCase
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 90b16c6d855c534fda1229f1afe7bc622cd0b7da in lucene-solr's branch refs/heads/branch_6x from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=90b16c6 ]

          SOLR-6246: add CHANGES entry

          Show
          jira-bot ASF subversion and git services added a comment - Commit 90b16c6d855c534fda1229f1afe7bc622cd0b7da in lucene-solr's branch refs/heads/branch_6x from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=90b16c6 ] SOLR-6246 : add CHANGES entry
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6c1a4b673a0b74d85d54593b76babe34bf543dbb in lucene-solr's branch refs/heads/master from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6c1a4b6 ]

          SOLR-6246: SolrSuggester.build() now throws SolrCoreState.CoreIsClosedException when a core reload/shutdown happens; add a random test lookup dictionary with configurable size; add

          {Analyzing,Blended}

          InfixSuggester reload/build tests; add a wrapped-exception expectThrows() variant to LuceneTestCase

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6c1a4b673a0b74d85d54593b76babe34bf543dbb in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6c1a4b6 ] SOLR-6246 : SolrSuggester.build() now throws SolrCoreState.CoreIsClosedException when a core reload/shutdown happens; add a random test lookup dictionary with configurable size; add {Analyzing,Blended} InfixSuggester reload/build tests; add a wrapped-exception expectThrows() variant to LuceneTestCase
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f9e36d9d76582e97103b29d2c4a4cf9d8e6fc1c6 in lucene-solr's branch refs/heads/master from Steve Rowe
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f9e36d9 ]

          SOLR-6246: add CHANGES entry

          Show
          jira-bot ASF subversion and git services added a comment - Commit f9e36d9d76582e97103b29d2c4a4cf9d8e6fc1c6 in lucene-solr's branch refs/heads/master from Steve Rowe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f9e36d9 ] SOLR-6246 : add CHANGES entry
          Hide
          steve_rowe Steve Rowe added a comment -

          Thanks jefferyyuan for testing!

          Show
          steve_rowe Steve Rowe added a comment - Thanks jefferyyuan for testing!
          Hide
          yuanyun.cn jefferyyuan added a comment - - edited

          Thanks Steve Rowe
          I am wondering is there any plan to also fix this issue in 6.4.x version?
          This fix is so valuable, without this we can't really use AnalyzingInfixSuggester - as we always reload the collections to update schema or config etc.

          And it takes time to release 6.5 - usually several(2 or 3) months.

          Show
          yuanyun.cn jefferyyuan added a comment - - edited Thanks Steve Rowe I am wondering is there any plan to also fix this issue in 6.4.x version? This fix is so valuable, without this we can't really use AnalyzingInfixSuggester - as we always reload the collections to update schema or config etc. And it takes time to release 6.5 - usually several(2 or 3) months.
          Hide
          t-rude Andreas Ravn added a comment -

          I'd also appreciate if this fix could be provided in 6.4.x. This bug is our critical impediment for a two-major-versions update.

          Show
          t-rude Andreas Ravn added a comment - I'd also appreciate if this fix could be provided in 6.4.x. This bug is our critical impediment for a two-major-versions update.
          Hide
          steve_rowe Steve Rowe added a comment -

          jefferyyuan, you yourself tested 6.4.1 and found that the fixes included in that release fixed your problem. Quoting the CHANGES.txt entry for this issue:

          Added tests to check that the changes in LUCENE-7564 and LUCENE-7670 enable AnalyzingInfixSuggester and BlendedInfixSuggester to play nicely with core reload.

          LUCENE-7564 and LUCENE-7670 contain the fixes for the problem identified in this issue, and they were included in Lucene/Solr 6.4.1.

          This issue contains tests that confirm the fix in the Solr context, and it wasn't ready until after the 6.4.1 release candidate voting was already underway, so couldn't be included.

          Here's what I included in the Solr 6.4.1 release announcement:

          AnalyzingInfixSuggester/BlendedInfixSuggester now work with core reload

          Show
          steve_rowe Steve Rowe added a comment - jefferyyuan , you yourself tested 6.4.1 and found that the fixes included in that release fixed your problem. Quoting the CHANGES.txt entry for this issue: Added tests to check that the changes in LUCENE-7564 and LUCENE-7670 enable AnalyzingInfixSuggester and BlendedInfixSuggester to play nicely with core reload. LUCENE-7564 and LUCENE-7670 contain the fixes for the problem identified in this issue, and they were included in Lucene/Solr 6.4.1. This issue contains tests that confirm the fix in the Solr context, and it wasn't ready until after the 6.4.1 release candidate voting was already underway, so couldn't be included. Here's what I included in the Solr 6.4.1 release announcement: AnalyzingInfixSuggester/BlendedInfixSuggester now work with core reload
          Hide
          yuanyun.cn jefferyyuan added a comment -

          This is great news. Thanks so much Steve Rowe for clarifying my questions.

          • I should do more search before asking here. My fault.
            I do read release notes for Solr 6.4.1 but not lucene 6.4.1 - which I should.
            Also I should have read the issue links this Jira depends on.
          Show
          yuanyun.cn jefferyyuan added a comment - This is great news. Thanks so much Steve Rowe for clarifying my questions. I should do more search before asking here. My fault. I do read release notes for Solr 6.4.1 but not lucene 6.4.1 - which I should. Also I should have read the issue links this Jira depends on.
          Hide
          dsmiley David Smiley added a comment -

          I was confused too! I recommend marking this as fixed for 6.4.1 and 6.5.0. If not both, then just 6.4.1. Of course this isn't a substitute for reading release notes but nonetheless I think adjusting the fix version is more true to the title of this issue being addressed at the 6.4.1 version.

          Show
          dsmiley David Smiley added a comment - I was confused too! I recommend marking this as fixed for 6.4.1 and 6.5.0. If not both, then just 6.4.1. Of course this isn't a substitute for reading release notes but nonetheless I think adjusting the fix version is more true to the title of this issue being addressed at the 6.4.1 version.
          Hide
          steve_rowe Steve Rowe added a comment - - edited

          I was confused too! I recommend marking this as fixed for 6.4.1 and 6.5.0 [...] I think adjusting the fix version is more true to the title of this issue being addressed at the 6.4.1 version.

          I think you're right - I added 6.4.1 as fix version. I'm leery of introducing an intentional difference between JIRA and CHANGES, but really the issue is that there should have been a Solr CHANGES note for the Lucene changes, but I didn't do that.

          Show
          steve_rowe Steve Rowe added a comment - - edited I was confused too! I recommend marking this as fixed for 6.4.1 and 6.5.0 [...] I think adjusting the fix version is more true to the title of this issue being addressed at the 6.4.1 version. I think you're right - I added 6.4.1 as fix version. I'm leery of introducing an intentional difference between JIRA and CHANGES, but really the issue is that there should have been a Solr CHANGES note for the Lucene changes, but I didn't do that.
          Hide
          alessandro.benedetti Alessandro Benedetti added a comment -

          i didn't have the time to investigate this issue in deep, but can this potentially affect also the DirectSolrSpellChecker?
          i will continue my investigations but I see a Solr cloud instance to fail to reload/restart, complaining about a lock in the index directory
          (
          ERROR [org.apache.solr.core.CoreContainer] (coreLoadExecutor-5-thread-6) Error creating core [core1]: Index locked for write for core core1: org.apache.solr.common.SolrException: Index locked for write for core core1

          on collection reload it just silentily times out...

          Show
          alessandro.benedetti Alessandro Benedetti added a comment - i didn't have the time to investigate this issue in deep, but can this potentially affect also the DirectSolrSpellChecker? i will continue my investigations but I see a Solr cloud instance to fail to reload/restart, complaining about a lock in the index directory ( ERROR [org.apache.solr.core.CoreContainer] (coreLoadExecutor-5-thread-6) Error creating core [core1] : Index locked for write for core core1: org.apache.solr.common.SolrException: Index locked for write for core core1 on collection reload it just silentily times out...
          Hide
          erickerickson Erick Erickson added a comment -

          Probably not. DirectSolrSpellChecker just looks in the index using (on a quick scan) IndexReader which doesn't hold a lock IIUC

          Show
          erickerickson Erick Erickson added a comment - Probably not. DirectSolrSpellChecker just looks in the index using (on a quick scan) IndexReader which doesn't hold a lock IIUC

            People

            • Assignee:
              steve_rowe Steve Rowe
              Reporter:
              varunthacker Varun Thacker
            • Votes:
              31 Vote for this issue
              Watchers:
              40 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development