Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-8930

Hang in tx.RemoteOperationMessage$RemoteOperationResponse.waitForRemoteResponse

    XMLWordPrintableJSON

Details

    Description

      In a race condition that 2 nodes can be deadlocked. The thread processing the other node's request can send a request to the other node when performing create in certain condition. If the other node is doing the same, both nodes can be deadlocked.

      "P2P message reader for rs-FullRegression04235130a0i3large-hydra-client-68(gemfire_host1_5646:5646)<ec><v1>:41001 shared ordered uid=3 port=38198" #65 daemon prio=10 os_prio=0 tid=0x00007fd3e4016800 nid=0x16b7 waiting on condition [0x00007fd3d3afa000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000000fdc0d4f0> (a java.util.concurrent.CountDownLatch$Sync)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
              at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
              at org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
              at org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)
              at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:802)
              at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:779)
              at org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:865)
              at org.apache.geode.internal.cache.tx.RemoteOperationMessage$RemoteOperationResponse.waitForRemoteResponse(RemoteOperationMessage.java:554)
              at org.apache.geode.internal.cache.tx.RemoteFetchVersionMessage$FetchVersionResponse.waitForResponse(RemoteFetchVersionMessage.java:229)
              at org.apache.geode.internal.cache.DistributedRegion.fetchRemoteVersionTag(DistributedRegion.java:3920)
              at org.apache.geode.internal.cache.TXEntryState.fetchRemoteVersionTag(TXEntryState.java:1062)
              at org.apache.geode.internal.cache.TXEntryState.basicPut(TXEntryState.java:1046)
              at org.apache.geode.internal.cache.TXState.txPutEntry(TXState.java:1375)
              at org.apache.geode.internal.cache.TXState.putEntry(TXState.java:1723)
              at org.apache.geode.internal.cache.TXState.putEntry(TXState.java:1707)
              at org.apache.geode.internal.cache.TXStateProxyImpl.putEntry(TXStateProxyImpl.java:683)
              at org.apache.geode.internal.cache.TXStateProxyImpl.putEntry(TXStateProxyImpl.java:673)
              at org.apache.geode.internal.cache.tx.RemotePutMessage.operateOnRegion(RemotePutMessage.java:640)
              at org.apache.geode.internal.cache.tx.RemoteOperationMessage.process(RemoteOperationMessage.java:207)
              at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
              at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:431)
              at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2063)
              at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1828)
              at org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$82/323166441.messageReceived(Unknown Source)
              at org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:936)
              at org.apache.geode.distributed.internal.membership.gms.GMSMembership.handleOrDeferMessage(GMSMembership.java:867)
              at org.apache.geode.distributed.internal.membership.gms.GMSMembership.processMessage(GMSMembership.java:1209)
              at org.apache.geode.distributed.internal.DistributionImpl$MyDCReceiver.messageReceived(DistributionImpl.java:826)
              at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:614)
              at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:661)
              at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3243)
              at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:2972)
              at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2778)
              at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1638)
      

      Attachments

        Issue Links

          Activity

            People

              eshu Eric Shu
              eshu Eric Shu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: