[GEODE-5653] Race between put of parallel WAN sender event and distributed clear causes off-heap orphan

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Component/s: offheap, regions, wan
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Ryan McMahon
• Created: 28/Aug/18 20:20
• Updated: 30/Aug/18 23:21
• Resolved: 30/Aug/18 23:20

Description

A race resulting in an off-heap orphan can occur with Parallel WAN when off-heap is enabled on the cache and user's data region. It is the result of the put occurring in two distinct steps without proper synchronization. The initial value of the region entry is Token.REMOVED_PHASE_1 (step 1) then later is replaced with actual GatewaySenderEvent (step 2). If a distributed clear is processed between these two steps, it can result in an orphan. More details on the race are described below.

The race is between:
Thread 1. A put of a GatewaySenderEvent containing an off-heap value to one of the WAN "shadow" region's BucketRegionQueue
and
Thread 2. A distributed clear on the BucketRegionQueue containing that value

The race occurs as follows:
Thread 1 (Put): Put results in a new region entry where the value is Token.REMOVED_PHASE1 and put it into the CustomEntryConcurrentHashMap owned by the AbstractRegionMap.
Thread 2 (Clear): Iterates through bucket region's segments and clears the entries. At this time the value is still Token.REMOVED_PHASE_1.
Thread 1 (Put): The Token.REMOVED_PHASE_1 is replaced with the actual GatewaySenderEvent in the region entry. However, the entry was removed via the clear above. When the entry is removed from the region, its off-heap is orphaned because it is no longer in the CustomEntryConcurrentHashMap.

Issue Links

• links to: GitHub Pull Request #2407