[SOLR-4116] Log Replay [recoveryExecutor-8-thread-1] - : java.io.EOFException

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 5.1
• Fix Version/s: 5.5, 6.0
• Component/s: SolrCloud

Environment

5.0.0.2012.11.28.10.42.06
Debian Squeeze, Tomcat 6, Sun Java 6, 10 nodes, 10 shards, rep. factor 2.

• Assignee: Erick Erickson
• Reporter: Markus Jelsma
• Created: 28/Nov/12 11:45
• Updated: 28/Mar/17 09:05
• Resolved: 28/Mar/17 09:05

Description

With SOLR-4032 fixed we see other issues when randomly taking down nodes (nicely via tomcat restart) while indexing a few million web pages from Hadoop. We do make sure that at least one node is up for a shard but due to recovery issues it may not be live.

2012-11-28 11:32:33,086 WARN [solr.update.UpdateLog] - [recoveryExecutor-8-thread-1] - : Starting log replay tlog{file=/opt/solr/cores/openindex_e/data/tlog/tlog.0000000000000000028 refcount=2} active=false starting pos=0
2012-11-28 11:32:41,873 ERROR [solr.update.UpdateLog] - [recoveryExecutor-8-thread-1] - : java.io.EOFException
        at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:151)
        at org.apache.solr.common.util.JavaBinCodec.readStr(JavaBinCodec.java:479)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:176)
        at org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCodec.java:374)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
        at org.apache.solr.common.util.JavaBinCodec.readArray(JavaBinCodec.java:451)
        at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:182)
        at org.apache.solr.update.TransactionLog$LogReader.next(TransactionLog.java:618)
        at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1198)
        at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1143)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Issue Links

• is related to: SOLR-8292 TransactionLog.next() does not honor contract and return null for EOF
• relates to: SOLR-4117 IO error while trying to get the size of the Directory