[GEODE-9150] partition region expiration uses extra memory on secondaries

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: expiration
• Labels: GeodeOperationAPI
• Assignee: Unassigned
• Reporter: Darrel Schneider
• Created: 13/Apr/21 23:22
• Updated: 22/Jun/21 16:16

Description

Partition region expiration usually drives all the expiration from the primary (the unusual case is when the pr also has evict destroy in which case expiration is done on primary and secondaries). But the secondaries still have all the expiration tasks (one for each entry which consumes about 112 bytes of memory per entry). When they fire on the secondary they go into a different pendingExpiry map owned by each bucket. They hang around in that map until the primary expiration happens or a secondary becomes primary. The pendingExpiry map also eats up some memory.

What we could do instead is no expiration scheduling on secondaries. All expiration would be driven from the primary and use the normal distribution to the secondaries. When a secondary becomes primary then it could kick off a background thread scheduling an expiry task for each entry. Currently when it becomes primary it processes the pending expiry map so that could be replaced with scheduling expiry. LocalRegion already has the code that supports rescheduling expiry since it is something users can dynamically change on an existing region. This would not only reduce the memory usage on secondaries but also simplify our expiration implementation.  PR eviction currently complicates our pr expiration implementation but if our pr eviction destroy was always distributed when the region had expiration, then we would not need any special expiration code to deal with evict destroy.