[OAK-1641] Mongo: Un-/CheckedExecutionException on replica-primary crash

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.19
• Fix Version/s: 1.1.8
• Component/s: mongomk

Environment

0.20-SNAPSHOT as of March 28, 2014

• Assignee: Marcel Reutegger
• Reporter: Stefan Egli
• Created: 28/Mar/14 14:38
• Updated: 07/Apr/15 09:09
• Resolved: 09/Mar/15 09:39

Description

Testing with a mongo replicaSet setup: 1 primary, 1 secondary and 1 secondary-arbiter-only.

Running a simple test which has 2 threads: a writer thread and a reader thread.

The following exception occurs when crashing mongo primary

com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: com.mongodb.MongoException$Network: Read operation to server localhost/127.0.0.1:12322 failed on database resilienceTest
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.getNode(DocumentNodeStore.java:593)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeState.hasChildNode(DocumentNodeState.java:164)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.hasChildNode(MemoryNodeBuilder.java:301)
	at org.apache.jackrabbit.oak.core.SecureNodeBuilder.hasChildNode(SecureNodeBuilder.java:299)
	at org.apache.jackrabbit.oak.plugins.tree.AbstractTree.hasChild(AbstractTree.java:267)
	at org.apache.jackrabbit.oak.core.MutableTree.getChild(MutableTree.java:147)
	at org.apache.jackrabbit.oak.util.TreeUtil.getTree(TreeUtil.java:171)
	at org.apache.jackrabbit.oak.jcr.delegate.NodeDelegate.getTree(NodeDelegate.java:865)
	at org.apache.jackrabbit.oak.jcr.delegate.NodeDelegate.getChild(NodeDelegate.java:339)
	at org.apache.jackrabbit.oak.jcr.session.NodeImpl$5.perform(NodeImpl.java:274)
	at org.apache.jackrabbit.oak.jcr.session.NodeImpl$5.perform(NodeImpl.java:1)
	at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:308)
	at org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:113)
	at org.apache.jackrabbit.oak.jcr.session.NodeImpl.addNode(NodeImpl.java:253)
	at org.apache.jackrabbit.oak.jcr.session.NodeImpl.addNode(NodeImpl.java:238)
	at org.apache.jackrabbit.oak.run.ReplicaCrashResilienceTest$1.run(ReplicaCrashResilienceTest.java:103)
	at java.lang.Thread.run(Thread.java:695)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.mongodb.MongoException$Network: Read operation to server localhost/127.0.0.1:12322 failed on database resilienceTest
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.find(MongoDocumentStore.java:267)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.find(MongoDocumentStore.java:234)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.readNode(DocumentNodeStore.java:802)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$3.call(DocumentNodeStore.java:596)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$3.call(DocumentNodeStore.java:1)
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
	... 19 more
Caused by: com.mongodb.MongoException$Network: Read operation to server localhost/127.0.0.1:12322 failed on database resilienceTest
	at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:253)
	at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:264)
	at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:264)
	at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216)
	at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:288)
	at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:273)
	at com.mongodb.DBCollection.findOne(DBCollection.java:728)
	at com.mongodb.DBCollection.findOne(DBCollection.java:670)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.findUncached(MongoDocumentStore.java:304)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore$1.call(MongoDocumentStore.java:270)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore$1.call(MongoDocumentStore.java:1)
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
	... 31 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:382)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:241)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:228)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:431)
	at java.net.Socket.connect(Socket.java:527)
	at com.mongodb.DBPort._open(DBPort.java:223)
	at com.mongodb.DBPort.go(DBPort.java:125)
	at com.mongodb.DBPort.call(DBPort.java:92)
	at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:244)
	... 46 more

Issue Links

• blocks: OAK-1453 MongoMK failover support for replica sets (esp. shards)