Uploaded image for project: 'Geode'
  1. Geode
  2. GEODE-9204

A not serializable exception can cause a ServerConnection thread to get stuck waiting for a reply from another member

    XMLWordPrintableJSON

Details

    Description

      A test case that reproduces it is:

      • a client get request is received in one server and sent to another server
      • the other server uses a CacheLoader to load the value
      • the CacheLoader throws an exception containing a non-serializable object
      • the reply attempts to serialize that exception but fails with NotSerializableException
      • the original server's ServerConnection thread gets stuck waiting for a reply that will never come

      Here is a stack trace showing the NotSerializableException:

      [severe 2018/03/20 14:30:27.793 PDT  <P2P message reader for elgreco(85544)<v11>:30177 unshared ordered uid=14 dom #1 port=53923> tid=0x5c] Uncaught exception processing  partitioned.GetMessage(prid=2 (name = "/data") processorId=0; posDup=false; key=0; callback arg=null; context=identity(elgreco(client:85552:loner):53907:fce35145:client,connection=2)
      org.apache.geode.InternalGemFireException: java.io.NotSerializableException: java.lang.Object
      	at org.apache.geode.internal.tcp.DirectReplySender.putOutgoing(DirectReplySender.java:76)
      	at org.apache.geode.distributed.internal.ReplyMessage.send(ReplyMessage.java:109)
      	at org.apache.geode.internal.cache.partitioned.PartitionMessage.sendReply(PartitionMessage.java:392)
      	at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:376)
      	at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:386)
      	at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:449)
      	at org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3872)
      	at org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3496)
      	at org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4693)
      	at org.apache.geode.distributed.internal.membership.jgroup.JGroupMembershipManager.processMessage(JGroupMembershipManager.java:2128)
      	at org.apache.geode.distributed.internal.membership.jgroup.JGroupMembershipManager.handleOrDeferMessage(JGroupMembershipManager.java:2037)
      	at org.apache.geode.distributed.internal.membership.jgroup.JGroupMembershipManager$MyDCReceiver.messageReceived(JGroupMembershipManager.java:647)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:804)
      	at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:835)
      	at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3932)
      	at org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3515)
      	at org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1827)
      	at org.apache.geode.internal.tcp.Connection.run(Connection.java:1702)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.NotSerializableException: java.lang.Object
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
      	at java.lang.Throwable.writeObject(Throwable.java:985)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
      	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
      	at java.lang.Throwable.writeObject(Throwable.java:985)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
      	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
      	at org.apache.geode.internal.InternalDataSerializer.writeSerializableObject(InternalDataSerializer.java:2352)
      	at org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2226)
      	at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:3179)
      	at org.apache.geode.distributed.internal.ReplyMessage.toData(ReplyMessage.java:305)
      	at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2402)
      	at org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1373)
      	at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:239)
      	at org.apache.geode.internal.tcp.DirectReplySender.putOutgoing(DirectReplySender.java:64)
      	... 18 more
      

      Here is where the ServerConnection thread is stuck:

      "ServerConnection on port 53895 Thread 1" #89 prio=5 os_prio=31 tid=0x00007f85d9001800 nid=0xcd07 runnable [0x000070000fddd000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
      	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
      	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
      	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
      	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
      	- locked <0x0000000780c82c28> (a java.lang.Object)
      	at org.apache.geode.internal.tcp.NIOMsgReader.readAtLeast(NIOMsgReader.java:45)
      	at org.apache.geode.internal.tcp.MsgReader.readHeader(MsgReader.java:38)
      	at org.apache.geode.internal.tcp.Connection.readAck(Connection.java:3365)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.readAcks(DirectChannel.java:567)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:499)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:310)
      	at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:696)
      	at org.apache.geode.distributed.internal.membership.jgroup.JGroupMembershipManager.directChannelSend(JGroupMembershipManager.java:2844)
      	at org.apache.geode.distributed.internal.membership.jgroup.JGroupMembershipManager.send(JGroupMembershipManager.java:3078)
      	at org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:79)
      	at org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3780)
      	at org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3821)
      	at org.apache.geode.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1957)
      	at org.apache.geode.internal.cache.partitioned.GetMessage.send(GetMessage.java:258)
      	at org.apache.geode.internal.cache.PartitionedRegion.getRemotely(PartitionedRegion.java:4990)
      	at org.apache.geode.internal.cache.PartitionedRegion.getFromBucket(PartitionedRegion.java:4096)
      	at org.apache.geode.internal.cache.PartitionedRegion.findObjectInSystem(PartitionedRegion.java:3459)
      	at org.apache.geode.internal.cache.PartitionedRegionDataView.findObject(PartitionedRegionDataView.java:65)
      	at org.apache.geode.internal.cache.PartitionedRegion.get(PartitionedRegion.java:3241)
      	at org.apache.geode.internal.cache.tier.sockets.command.Get70.getValueAndIsObject(Get70.java:271)
      	at org.apache.geode.internal.cache.tier.sockets.command.Get70.cmdExecute(Get70.java:162)
      	at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:182)
      	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:789)
      	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:920)
      	at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1165)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:579)
      	at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        Activity

          People

            kaslami Kamilla Aslami
            bschuchardt Bruce J Schuchardt
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: