[GEODE-2293] The AckReaderThread incorrectly shuts down when an IllegalStateException is thrown while releasing an off-heap object

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: wan
• Labels: regression
• Assignee: Jason Huynh
• Reporter: Barrett Oglesby
• Created: 10/Jan/17 19:54
• Updated: 08/Feb/17 22:21
• Resolved: 18/Jan/17 22:35

Description

The regression test run showed the following severe message:

[severe 2017/01/07 09:14:40.789 UTC bridgegemfire_1_1_rs-QueuesBTTest-2017-01-06-14-07-15-client-12_3912 <AckReaderThread for : Event Processor for GatewaySender_sender_bridgeds_1_to_bridgeds_2_0> tid=0x97] Stopping the processor because the following exception occurred while processing a batch:
java.lang.IllegalStateException: It looks like off heap memory @7f33a8000238 was already freed. rawBits=0 history=null
	at org.apache.geode.internal.offheap.OffHeapStoredObject.release(OffHeapStoredObject.java:675)
	at org.apache.geode.internal.offheap.OffHeapStoredObject.release(OffHeapStoredObject.java:659)
	at org.apache.geode.internal.offheap.OffHeapStoredObject.release(OffHeapStoredObject.java:373)
	at org.apache.geode.internal.offheap.OffHeapHelper.releaseAndTrackOwner(OffHeapHelper.java:138)
	at org.apache.geode.internal.cache.wan.GatewaySenderEventImpl.release(GatewaySenderEventImpl.java:1213)
	at org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.remove(ParallelGatewaySenderQueue.java:1096)
	at org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.remove(ParallelGatewaySenderQueue.java:1531)
	at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.eventQueueRemove(AbstractGatewaySenderEventProcessor.java:231)
	at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.handleSuccessBatchAck(AbstractGatewaySenderEventProcessor.java:981)
	at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher$AckReaderThread.run(GatewaySenderEventRemoteDispatcher.java:636)

This exception shows that the AckReaderThread was processing a successful batch acknowledgement, and an IllegalStateException was thrown while releasing a GatewaySenderEventImpl from off-heap memory. This caused the AckReaderThread to shut down. It looks like the GatewaySenderEventImpl had already been released and is being released again. I'm not sure how the GatewaySenderEventImpl got into this state, but the AckReaderThread should not shut down because of this IllegalStateException.

The code in question is in the finally block of ParallelGatewaySenderQueue.remove:

} finally {
  event.release();
}