Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-12487

Possible corruption in tlog after restart of Solr

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 6.5.1
    • Fix Version/s: None
    • Component/s: SolrCloud, update
    • Labels:
      None
    • Environment:

      Windows server 2012

      12 CPU

      96GB RAM

      -DzkClientTimeout=45000-DzkRun
      --Xms16g-Xmx16g-Xss256k

      Description

      After restarting Solr we encountered issues replaying tlogs from one of our six shards. The other shards were up and running after a couple hours of replay (due to total tlog of 20gb which we are going to change from "on request" to maxDocs/maxTime). 

      The log first states warnings about "Unexpected log entry or corrupt log" and then we get the following error message:

       

      java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [VCloud_shard1_replica1]
      at java.util.concurrent.FutureTask.report(Unknown Source)
      at java.util.concurrent.FutureTask.get(Unknown Source)
      at org.apache.solr.core.CoreContainer.lambda$load$6(CoreContainer.java:581)
      at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: org.apache.solr.common.SolrException: Unable to create core [VCloud_shard1_replica1]
      at org.apache.solr.core.CoreContainer.create(CoreContainer.java:933)
      at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:553)
      at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
      ... 5 more
      Caused by: org.apache.solr.common.SolrException: Error Instantiating Update Handler, solr.DirectUpdateHandler2 failed to instantiate org.apache.solr.update.UpdateHandler
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:965)
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:831)
      at org.apache.solr.core.CoreContainer.create(CoreContainer.java:918)
      ... 7 more
      Caused by: org.apache.solr.common.SolrException: Error Instantiating Update Handler, solr.DirectUpdateHandler2 failed to instantiate org.apache.solr.update.UpdateHandler
      at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:775)
      at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:823)
      at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1071)
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:936)
      ... 9 more
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
      at java.lang.reflect.Constructor.newInstance(Unknown Source)
      at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:761)
      ... 12 more
      Caused by: java.lang.StackOverflowError
      at org.apache.solr.common.util.JavaBinCodec.readVInt(JavaBinCodec.java:998)
      at org.apache.solr.common.util.JavaBinCodec.readMap(JavaBinCodec.java:612)
      at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:298)
      etc......

      ...

      After this happens it won't do any more work on this shard and the main page in solr displays:
      "VCloud_shard1_replica1: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error Instantiating Update Handler, solr.DirectUpdateHandler2 failed to instantiate org.apache.solr.update.UpdateHandler"

       

      Based on the execption and the huge repeating StackOverflowError we believe the tlog is corrupt and missing e.g. an end tag. 

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              larsg Lars Gjestang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: