Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11378

Solr can leak transaction logs if a commit happens during unload

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      I have a test called AutoscalingHistoryHandlerTest in the feature/autoscaling branch that fails fairly frequently due to the ObjectReleaseTracker complaining about TransactionLog not being closed. This happens because the solr core in question is being moved and is therefore unloaded but an auto-commit was in progress during this time. The commit does not succeed because a new searcher could not be opened and I think that causes the tlog reference to be leaked.

      It should be possible to isolate this problem into a smaller test by continuously indexing documents with a commitWithin and unloading the core in the middle of indexing.

      Here are the relevant stack traces:

      21106 ERROR (commitScheduler-48-thread-1) [n:127.0.0.1:54191_solr c:.system s:shard1 r:core_node5 x:.system_shard1_replica_n2] o.a.s.u.CommitTracker auto commit error...:org.apache.solr.common.SolrException: Error opening new searcher
      	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2076)
      	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2196)
      	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1933)
      	at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:710)
      	at org.apache.solr.update.CommitTracker.run(CommitTracker.java:222)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.solr.common.SolrException: openNewSearcher called on closed core
      	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2063)
      	... 11 more
      
      java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [TransactionLog]
      org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.update.TransactionLog
      	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
      	at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:190)
      	at org.apache.solr.update.UpdateLog.newTransactionLog(UpdateLog.java:446)
      	at org.apache.solr.update.UpdateLog.ensureLog(UpdateLog.java:1259)
      	at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
      	at org.apache.solr.update.UpdateLog.add(UpdateLog.java:517)
      	at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:347)
      	at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:266)
      	at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:216)
      	at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
      	at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
      	at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:991)
      	at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1207)
      	at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:753)
      	at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
      	at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
      	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:188)
      	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:144)
      	at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:311)
      	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
      	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:130)
      	at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:276)
      	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:256)
      	at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:178)
      	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:195)
      	at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:108)
      	at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
      	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
      	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
      	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
      	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2484)
      	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720)
      	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              shalin Shalin Shekhar Mangar
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: