[SOLR-6583] Resuming connection with ZooKeeper causes log replay

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.10.1
• Fix Version/s: 5.1, 6.0
• Component/s: SolrCloud
• Assignee: Shalin Shekhar Mangar
• Reporter: Shalin Shekhar Mangar
• Created: 03/Oct/14 10:20
• Updated: 09/May/16 18:54
• Resolved: 20/Apr/15 13:39

Description

If a node is partitioned from ZooKeeper for an extended period of time then upon resuming connection, the node re-registers itself causing recoverFromLog() method to be executed which fails with the following exception:

8091124 [Thread-71] ERROR org.apache.solr.update.UpdateLog  – Error inspecting tlog tlog{file=/home/ubuntu/shalin-lusolr/solr/example/solr/collection_5x3_shard5_replica3/data/tlog/tlog.0000000000000009869 refcount=2}
java.nio.channels.ClosedChannelException
        at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:678)
        at org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:784)
        at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
        at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:125)
        at java.io.InputStream.read(InputStream.java:101)
        at org.apache.solr.update.TransactionLog.endsWithCommit(TransactionLog.java:218)
        at org.apache.solr.update.UpdateLog.recoverFromLog(UpdateLog.java:800)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:834)
        at org.apache.solr.cloud.ZkController$1.command(ZkController.java:271)
        at org.apache.solr.common.cloud.ConnectionManager$1$1.run(ConnectionManager.java:166)
8091125 [Thread-71] ERROR org.apache.solr.update.UpdateLog  – Error inspecting tlog tlog{file=/home/ubuntu/shalin-lusolr/solr/example/solr/collection_5x3_shard5_replica3/data/tlog/tlog.0000000000000009870 refcount=2}
java.nio.channels.ClosedChannelException
        at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:678)
        at org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:784)
        at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
        at org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:125)
        at java.io.InputStream.read(InputStream.java:101)
        at org.apache.solr.update.TransactionLog.endsWithCommit(TransactionLog.java:218)
        at org.apache.solr.update.UpdateLog.recoverFromLog(UpdateLog.java:800)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:834)
        at org.apache.solr.cloud.ZkController$1.command(ZkController.java:271)
        at org.apache.solr.common.cloud.ConnectionManager$1$1.run(ConnectionManager.java:166)

This is because the recoverFromLog uses transaction log references that were collected at startup and are no longer valid.

We shouldn't even be running recoverFromLog code for ZK re-connect.

Issue Links

• is related to: SOLR-7338 A reloaded core will never register itself as active after a ZK session expiration