[GEODE-8930] Hang in tx.RemoteOperationMessage$RemoteOperationResponse.waitForRemoteResponse

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.12.1, 1.13.2, 1.14.0
• Component/s: transactions
• Labels: pull-request-available
• Assignee: Eric Shu
• Reporter: Eric Shu
• Created: 09/Feb/21 16:59
• Updated: 26/Mar/21 22:25
• Resolved: 10/Feb/21 16:58

Description

In a race condition that 2 nodes can be deadlocked. The thread processing the other node's request can send a request to the other node when performing create in certain condition. If the other node is doing the same, both nodes can be deadlocked.

"P2P message reader for rs-FullRegression04235130a0i3large-hydra-client-68(gemfire_host1_5646:5646)<ec><v1>:41001 shared ordered uid=3 port=38198" #65 daemon prio=10 os_prio=0 tid=0x00007fd3e4016800 nid=0x16b7 waiting on condition [0x00007fd3d3afa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000fdc0d4f0> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
        at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:802)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:779)
        at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:865)
        at org.apache.geode.internal.cache.tx.RemoteOperationMessage$RemoteOperationResponse.waitForRemoteResponse(RemoteOperationMessage.java:554)
        at org.apache.geode.internal.cache.tx.RemoteFetchVersionMessage$FetchVersionResponse.waitForResponse(RemoteFetchVersionMessage.java:229)
        at org.apache.geode.internal.cache.DistributedRegion.fetchRemoteVersionTag(DistributedRegion.java:3920)
        at org.apache.geode.internal.cache.TXEntryState.fetchRemoteVersionTag(TXEntryState.java:1062)
        at org.apache.geode.internal.cache.TXEntryState.basicPut(TXEntryState.java:1046)
        at org.apache.geode.internal.cache.TXState.txPutEntry(TXState.java:1375)
        at org.apache.geode.internal.cache.TXState.putEntry(TXState.java:1723)
        at org.apache.geode.internal.cache.TXState.putEntry(TXState.java:1707)
        at org.apache.geode.internal.cache.TXStateProxyImpl.putEntry(TXStateProxyImpl.java:683)
        at org.apache.geode.internal.cache.TXStateProxyImpl.putEntry(TXStateProxyImpl.java:673)
        at org.apache.geode.internal.cache.tx.RemotePutMessage.operateOnRegion(RemotePutMessage.java:640)
        at org.apache.geode.internal.cache.tx.RemoteOperationMessage.process(RemoteOperationMessage.java:207)
        at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
        at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:431)
        at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2063)
        at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1828)
        at org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$82/323166441.messageReceived(Unknown Source)
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:936)
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership.handleOrDeferMessage(GMSMembership.java:867)
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership.processMessage(GMSMembership.java:1209)
        at org.apache.geode.distributed.internal.DistributionImpl$MyDCReceiver.messageReceived(DistributionImpl.java:826)
        at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:614)
        at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:661)
        at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3243)
        at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:2972)
        at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2778)
        at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1638)

Issue Links

• causes: GEODE-8947 Performance degradations due to GEODE-8930
• links to: GitHub Pull Request #6018