[GEODE-2027] Gateway events being processed by the ParallelQueueRemovalMessage in the secondary at the time of GII may not be removed from the queue successfully

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0-incubating
• Component/s: wan
• Assignee: Barrett Oglesby
• Reporter: Barrett Oglesby
• Created: 21/Oct/16 19:38
• Updated: 28/Dec/16 00:06
• Resolved: 07/Nov/16 21:13

Description

This is the same as Trac ticket 49196.

I see two cases that cause this issue (so far). In both cases the member is starting and GIIing the bucket region, and the entry in question is included in the GII response.

In case one:

• the bucket region is not found in the ParallelGatewaySenderQueue.put path, so the event is added to the temp queue
• the bucket region is found in the ParallelQueueRemovalMessage.process path and contains the key so it is removed from it
• the event is added back into the bucket region from the temp queue

The event is added to the temp queue:

[warning 2016/10/20 01:25:07.715 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 <P2P message reader for ip-10-32-109-74(bridgegemfire_1_2_ip-10-32-109-74_25713:25713)<ec><v1>:1026 unshared ordered uid=75 dom #1 port=57564> tid=0x82] XXX ParallelGatewaySenderQueue.put adding to tempQueue brq=null; key=33283; value=SenderEventImpl[id=EventID[threadID=0x10007|5;sequenceID=177;bucketId=7];action=2;operation=DESTROY;region=/testRegion1;key=Object_2934;value=;valueIsObject=1;numberOfParts=8;callbackArgument=GatewaySenderEventCallbackArgument [originalCallbackArg=null;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=false;creationTime=1476926707714;shadowKey= 33283;timeStamp=1476926707709;acked=false;dispatched=false]

The event exists in the bucket region and is removed:

[warning 2016/10/20 01:25:10.265 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 <Pooled Message Processor 1> tid=0x63] XXX ParallelQueueRemovalMessage.process bucket is not initialized but contains key destroying from bucket name=/__PR/_B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7; key=33283

The event is loaded from the temp queue into the bucket region:

[warning 2016/10/20 01:25:10.432 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 <ResourceManagerRecoveryThread 0> tid=0x4f] XXX AbstractBucketRegionQueue.loadEventsFromTempQueue _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7 adding to queue key=33283 -> SenderEventImpl[id=EventID[threadID=0x10007|5;sequenceID=177;bucketId=7];action=2;operation=DESTROY;region=/testRegion1;key=Object_2934;value=;valueIsObject=1;numberOfParts=8;callbackArgument=GatewaySenderEventCallbackArgument [originalCallbackArg=null;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=true;creationTime=1476926707714;shadowKey= 33283;timeStamp=1476926707709;acked=false;dispatched=false]

The bucket region contains the key right before it is available for use:

[warning 2016/10/20 01:25:10.790 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 <ResourceManagerRecoveryThread 0> tid=0x4f] XXX BucketRegionQueue.cleanUpDestroyedTokensAndMarkGIIComplete _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7 keySet4=[33283, 33079, 34591, 35071, 35119, 34879, 34447, 34891, 34843, 35083, 33295, 34387, 34339, 32743, 35131, 35095, 35059, 34603, 34471, 34567, 34375, 32983, 32875, 34399, 34867, 34831, 34759, 34675, 35107, 35047, 34723, 34495, 34507, 34459, 34699, 34735]

In case two:

• the bucket region is not found in the ParallelGatewaySenderQueue.put path, so the event is added to the temp queue
• the bucket region is found in the ParallelQueueRemovalMessage.process path, but doesn't contain the key so it is removed from the temp queue
• the event is removed from the temp queue but is also in the bucket region

The event is added to the temp queue:

[warning 2016/10/19 22:55:27.743 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 <P2P message reader for ip-10-32-109-74(bridgegemfire_1_3_ip-10-32-109-74_24716:24716)<ec><v1>:1028 unshared ordered uid=62 dom #1 port=56962> tid=0x68] XXX ParallelGatewaySenderQueue.put adding to tempQueue brq=null; key=31555; value=SenderEventImpl[id=EventID[threadID=0x10007|7;sequenceID=108;bucketId=7];action=0;operation=CREATE;region=/testRegion1;key=Object_10236;value=util.ValueHolder [myVersion=util.ValueHolder, myValue=<java.lang.Long, value 10236>, extraObject=<[B Size 70, componentType byte>, modVal=null];valueIsObject=1;numberOfParts=9;callbackArgument=GatewaySenderEventCallbackArgument [originalCallbackArg=July;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=false;creationTime=1476917727742;shadowKey= 31555;timeStamp=1476917727723;acked=false;dispatched=false]

The event doesn't exist in the bucket region and is removed from the temp queue:

[warning 2016/10/19 22:55:30.047 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 <Pooled Message Processor 2> tid=0x64] XXX ParallelQueueRemovalMessage.process bucket is not initialized and does not contain key destroying from tempQueue name=/__PR/_B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7; key=31555

[warning 2016/10/19 22:55:30.048 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 <Pooled Message Processor 2> tid=0x64] XXX ParallelQueueRemovalMessage.destroyFromTempQueue destroyed qPR=sender_bridgeds_1_to_bridgeds_2_PARALLEL_GATEWAY_SENDER_QUEUE; bucketId=7; keyToRemove=31555

The bucket region contains the key right before it is available for use:

[warning 2016/10/19 22:55:30.239 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 <ResourceManagerRecoveryThread 0> tid=0x50] XXX BucketRegionQueue.cleanUpDestroyedTokensAndMarkGIIComplete _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7 keySet4=[33079, 31807, 32215, 33043, 32815, 31555, 32323, 32695, 32455, 32539, 32575, 32959, 32287, 32923, 32359, 32995, 33031, 32623, 33067, 32647, 32395, 32479, 33007, 32779, 32551, 32515, 32587, 32251, 32767, 32899, 32383, 32635, 32791, 32731, 32311, 32827, 32275, 32599, 32851, 32803, 32503, 32527, 32347, 32911, 32371, 32335]