[SOLR-4188] Transaction log corruption

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 6.0
• Fix Version/s: 6.0
• Component/s: SolrCloud

Environment

5.0.0.2012.12.14.10.53.04

• Assignee: Unassigned
• Reporter: Markus Jelsma
• Created: 14/Dec/12 10:40
• Updated: 09/May/16 18:44
• Resolved: 02/Oct/13 09:39

Description

We've updates the 10 node test cluster with today's check out. Just like some of my other recent issues this one is also restricted to a single node, the same node that's throwing SOLR-4187 but i'm not very sure both are the same issue. This one seems to be caused only when the node is trying to replicate.

Tomcat has never died ungracefully and restart is graceful.

2012-12-14 10:35:23,789 ERROR [solr.cloud.RecoveryStrategy] - [RecoveryThread] - : Corrupt tlog - ignoring. core=shard_f:org.apache.solr.common.SolrException: incref on a closed log: tlog{file=/opt/solr/cores/shard_f/data/tlog/tlog.0000000000000000084 refcount=1}
        at org.apache.solr.update.TransactionLog.incref(TransactionLog.java:492)
        at org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:989)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:259)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)

The trace below seems to accompany the one above. This one is being logged some 10 times in a row, exactly the same timestamp.

2012-12-14 10:40:36,152 ERROR [solr.update.UpdateLog] - [RecoveryThread] - : Exception reading versions from log
java.nio.channels.ClosedChannelException
        at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:603)
        at org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:752)
        at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
        at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:71)
        at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
        at org.apache.solr.update.TransactionLog$ReverseReader.<init>(TransactionLog.java:670)
        at org.apache.solr.update.TransactionLog.getReverseReader(TransactionLog.java:573)
        at org.apache.solr.update.UpdateLog$RecentUpdates.update(UpdateLog.java:912)
        at org.apache.solr.update.UpdateLog$RecentUpdates.access$000(UpdateLog.java:855)
        at org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1005)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:259)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)