[SOLR-6776] Data lost when use SoftCommit and TLog

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.10
• Fix Version/s: 4.10.3
• Labels: softCommit updateLog
• Assignee: Yonik Seeley
• Reporter: jefferyyuan
• Created: 21/Nov/14 21:39
• Updated: 02/Dec/14 16:49
• Resolved: 02/Dec/14 16:49

Description

We enabled update log and change autoCommit to some bigger value 10 mins.

After restart, we push one doc with softCommit=true
http://localhost:8983/solr/update?stream.body=<add><doc><field name="id">id1</field></doc></add>&softCommit=true

Then we kill the java process after a min.

After restart, Tlog failed to replay with following exception, and there is no data in solr.
6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog û Failure to open existing log file (non fatal) E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
log\tlog.0000000000000000000:org.apache.solr.common.SolrException: java.io.EOFException
at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:181)
at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:134)
at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)
at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:100)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:835)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
at org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:159)
... 19 more

Check the code: seems this is related with: org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it sets changesSinceCommit=false(even we are using softCommit)

So in finish, updateLog.finish will not be called.
public void finish() throws IOException {
if (changesSinceCommit && updateHandler.getUpdateLog() != null)

{ updateHandler.getUpdateLog().finish(null); }

super.finish();
}

To fix this issue: I have to change RunUpdateProcessor.processCommit like below:
if (!cmd.softCommit)

{ changesSinceCommit = false; }