[GEODE-7920] Geode UDP INT thread found processing cache operations

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.6
• Fix Version/s: 1.12.6, 1.13.0
• Component/s: membership, messaging
• Labels: pull-request-available
• Assignee: Bill Burcham
• Reporter: Bruce J Schuchardt
• Created: 27/Mar/20 20:47
• Updated: 11/Dec/21 03:01
• Resolved: 03/Apr/20 15:28

Description

While looking into another problem in a test run with disable-tcp=true I found this JGroups thread processing a cache operation in-line. These threads should never process cache ops. They exist just to read messages and hand them off to an Executor.

Fixing this should improve UDP messaging performance somewhat.

"Geode UDP INT-2,rs-Awesome-781-1145-1a0i32xlarge-hydra-client-4-27004" #54 daemon prio=5 os_prio=0 tid=0x00007f202c05b800 nid=0x778d runnable [0x00007f20b0bb6000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	- locked <0x00000000c292f560> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.readByte(DataInputStream.java:265)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:240)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:163)
	at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
	at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
	at com.sun.proxy.$Proxy53.increment(Unknown Source)
	at hydra.blackboard.AnySharedCountersImpl.increment(AnySharedCountersImpl.java:159)
	at util.AbstractListener.incrementAfterDestroyCounters(AbstractListener.java:450)
	at event.ETListener.afterDestroy(ETListener.java:88)
	at org.apache.geode.internal.cache.EnumListenerEvent$AFTER_DESTROY.dispatchEvent(EnumListenerEvent.java:178)
	at org.apache.geode.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:8242)
	at org.apache.geode.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:6952)
	at org.apache.geode.internal.cache.LocalRegion.invokeDestroyCallbacks(LocalRegion.java:6760)
	at org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2443)
	at org.apache.geode.internal.cache.entries.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:164)
	at org.apache.geode.internal.cache.LocalRegion.basicDestroyPart2(LocalRegion.java:6701)
	at org.apache.geode.internal.cache.map.RegionMapDestroy.removeEntryOrLeaveTombstone(RegionMapDestroy.java:509)
	at org.apache.geode.internal.cache.map.RegionMapDestroy.retryRemoveWithTombstone(RegionMapDestroy.java:373)
	- locked <0x00000000fe71d6e0> (a org.apache.geode.internal.cache.entries.VersionedStatsRegionEntryOffHeapStringKey2)
	at org.apache.geode.internal.cache.map.RegionMapDestroy.checkTombstoneAndConcurrency(RegionMapDestroy.java:202)
	at org.apache.geode.internal.cache.map.RegionMapDestroy.destroy(RegionMapDestroy.java:147)
	at org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:980)
	at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6490)
	at org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6464)
	at org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:58)
	at org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6415)
	at org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1720)
	at org.apache.geode.internal.cache.DestroyOperation$DestroyMessage.operateOnRegion(DestroyOperation.java:88)
	at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1208)
	at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1110)
	at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
	at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:431)
	at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2072)
	at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1847)
	at org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$74/1548842157.messageReceived(Unknown Source)
	at org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:931)

This can be fixed by re-enabling the commented out code in DistributionMessage.isPreciousThread(). There are two thread-name checks already in that method that need to be retained as well, for the unicast-reader thread and the multicast-reader thread.

Issue Links

• links to: GitHub Pull Request #4893
• links to: GitHub Pull Request #7018