[GEODE-1915] Registering instantiators can cause a deadlock with gateways

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0-incubating
• Component/s: wan
• Assignee: Dan Smith
• Reporter: Dan Smith
• Created: 21/Sep/16 00:53
• Updated: 25/Oct/16 07:18
• Resolved: 22/Sep/16 20:49

Description

If two WAN sites are connected bidirectionally, registering an instantiator in one of the sites can cause a deadlock.

The issue is that when an instantiator is registered, a message is sent synchronously from one site to the other, while holding a static lock on the InternalInstantiator class. Unfortunately, when the second site receives the registration, it tries to send it back to the first site. In the first site, the registeration message then is stuck trying to get the same lock.

"ServerConnection on port 28517 Thread 4" #80 daemon prio=5 os_prio=0 tid=0x00007fce78007000 nid=0xc48a runnable [0x00007fce377f7000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.geode.internal.cache.tier.sockets.Message.fetchHeader(Message.java:693)
	at org.apache.geode.internal.cache.tier.sockets.Message.readHeaderAndPayload(Message.java:710)
	at org.apache.geode.internal.cache.tier.sockets.Message.read(Message.java:661)
	at org.apache.geode.internal.cache.tier.sockets.Message.recv(Message.java:1103)
	- locked <0x00000000fb3aec38> (a java.nio.HeapByteBuffer)
	at org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:171)
	at org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:388)
	at org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:272)
	- locked <0x00000000fb3ad430> (a org.apache.geode.cache.client.internal.ConnectionImpl)
	at org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:328)
	at org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:937)
	at org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:155)
	at org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:711)
	at org.apache.geode.cache.client.internal.RegisterInstantiatorsOp.execute(RegisterInstantiatorsOp.java:49)
	at org.apache.geode.internal.cache.PoolManagerImpl.allPoolsRegisterInstantiator(PoolManagerImpl.java:227)
	at org.apache.geode.internal.InternalInstantiator.sendRegistrationMessageToServers(InternalInstantiator.java:219)
	at org.apache.geode.internal.InternalInstantiator._register(InternalInstantiator.java:174)
	- locked <0x00000000fb3ad678> (a java.lang.Class for org.apache.geode.internal.InternalInstantiator)
	at org.apache.geode.internal.InternalInstantiator.register(InternalInstantiator.java:310)
	at org.apache.geode.internal.cache.tier.sockets.command.RegisterInstantiators.cmdExecute(RegisterInstantiators.java:100)
	at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:145)
	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1180)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
	at java.lang.Thread.run(Thread.java:745)

"ServerConnection on port 28517 Thread 3" #78 daemon prio=5 os_prio=0 tid=0x00007fce78005000 nid=0xc487 waiting for monitor entry [0x00007fce379fa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.geode.internal.InternalInstantiator._register(InternalInstantiator.java:117)
	- waiting to lock <0x00000000fb3ad678> (a java.lang.Class for org.apache.geode.internal.InternalInstantiator)
	at org.apache.geode.internal.InternalInstantiator.register(InternalInstantiator.java:310)
	at org.apache.geode.internal.cache.tier.sockets.command.RegisterInstantiators.cmdExecute(RegisterInstantiators.java:100)
	at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:145)
	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:913)
	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1180)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:546)
	at java.lang.Thread.run(Thread.java:745)