Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6776

Data lost when use SoftCommit and TLog

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.10
    • Fix Version/s: 4.10.3
    • Component/s: None

      Description

      We enabled update log and change autoCommit to some bigger value 10 mins.

      After restart, we push one doc with softCommit=true
      http://localhost:8983/solr/update?stream.body=<add><doc><field name="id">id1</field></doc></add>&softCommit=true

      Then we kill the java process after a min.

      After restart, Tlog failed to replay with following exception, and there is no data in solr.
      6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog รป Failure to open existing log file (non fatal) E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
      log\tlog.0000000000000000000:org.apache.solr.common.SolrException: java.io.EOFException
      at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:181)
      at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
      at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:134)
      at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)
      at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:100)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
      at java.lang.reflect.Constructor.newInstance(Unknown Source)
      at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
      at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:835)
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
      at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
      at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
      at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.io.EOFException
      at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
      at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
      at org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
      at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:159)
      ... 19 more

      Check the code: seems this is related with: org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it sets changesSinceCommit=false(even we are using softCommit)

      So in finish, updateLog.finish will not be called.
      public void finish() throws IOException {
      if (changesSinceCommit && updateHandler.getUpdateLog() != null)

      { updateHandler.getUpdateLog().finish(null); }

      super.finish();
      }

      To fix this issue: I have to change RunUpdateProcessor.processCommit like below:
      if (!cmd.softCommit)

      { changesSinceCommit = false; }

        Attachments

          Activity

            People

            • Assignee:
              yseeley@gmail.com Yonik Seeley
              Reporter:
              yuanyun.cn jefferyyuan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: