Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-6929

In the case of a ConcurrentCacheModificationException that occurs while processing a RemotePutMessage, the reply is attempted to be sent twice causing an InternalGemFireError

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.10.0
    • Component/s: messaging
    • Labels:
      None

      Description

      The RemotePutMessage operateOnRegion method attempts to sendReply twice if result is false:

      try {
        result = r.getDataView().putEntry(event, this.ifNew, this.ifOld, this.expectedOldValue,
            this.requireOldValue, this.lastModified, true);
      
        if (!this.result) { // make sure the region hasn't gone away
          r.checkReadiness();
          if (!this.ifNew && !this.ifOld) {
            // no reason to be throwing an exception, so let's retry
            RemoteOperationException ex = new RemoteOperationException(
                "unable to perform put, but operation should not fail");
      1 ->  sendReply(getSender(), getProcessorId(), dm, new ReplyException(ex), r, startTime);
          }
        }
      ...
      if (sendReply) {
      2->  sendReply(getSender(), getProcessorId(), dm, null, r, startTime, event);
      }
      

      This causes this fatal InternalGemFireError:

      [fatal 2019/06/28 15:33:01.005 PDT <P2P message reader for 192.168.1.2(gateway-ny-proxy-1:77395)<v85>:41003 unshared ordered uid=12 dom #1 port=58836> tid=0x4c] Uncaught exception processing tx.RemotePutMessage(regionPath=/TradeDateCalendar; sender=192.168.1.2(gateway-ny-proxy-1:77395)<v85>:41003; recipients=[null]; processorId=0; key=87; value=(5 bytes); callback=GatewaySenderEventCallbackArgument [originalCallbackArg=null;originatingSenderId=2;recipientGatewayReceivers={1}]; op=UPDATE; bridgeContext=identity(192.168.1.2(gateway-ln-data-1:77388)<v43>:41002,connection=1; eventId=EventID[id=31 bytes;threadID=0x30001|1;sequenceID=731]; ifOld=false; ifNew=false; op=UPDATE; hadOldValue=false; deserializationPolicy=LAZY; hasDelta=false; sendDelta=false; isDeltaApplied=false ,distTx=false)
      org.apache.geode.InternalGemFireError: Trying to reply twice to a message
      	at org.apache.geode.internal.Assert.throwError(Assert.java:89)
      	at org.apache.geode.internal.Assert.assertTrue(Assert.java:107)
      	at org.apache.geode.internal.tcp.DirectReplySender.putOutgoing(DirectReplySender.java:55)
      	at org.apache.geode.internal.cache.tx.RemotePutMessage$PutReplyMessage.send(RemotePutMessage.java:791)
      	at org.apache.geode.internal.cache.tx.RemotePutMessage.sendReply(RemotePutMessage.java:675)
      	at org.apache.geode.internal.cache.tx.RemoteOperationMessage.process(RemoteOperationMessage.java:266)
      	at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:369)
      	at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:425)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2891)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:2571)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager.access$1400(ClusterDistributionManager.java:110)
      	at org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.messageReceived(ClusterDistributionManager.java:3430)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1108)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1027)
      	at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:407)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:701)
      	at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:702)
      	at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3427)
      	at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:3164)
      	at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2959)
      	at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1743)
      	at org.apache.geode.internal.tcp.Connection.run(Connection.java:1579)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                barry.oglesby Barry Oglesby
                Reporter:
                barry.oglesby Barry Oglesby
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m