Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
If a partitioned region is configured with evict destroy and with distributed expiration, then the expiration action is no longer distributed but is a local operation performed independently on the primary and each secondary. This can lead to partitioned regions that give inconsistent read results. For example a get op can be sent to primary or any secondary so it may exist on one but not another.
Changing these distributed expiration actions to be local was a choice made about 11 years ago to deal with the fact that evict destroy does local destroys that can also cause the cache to be inconsistent. If the evict destroy was done on the primary then it caused the secondary entries to never expire (since normally all expiration on a pr is driven from the primary).
But even though we needed to make sure expiration still happened, even if the entry was local destroyed on the primary, we could have done it and still had expiration be distributed. We could have sent a message to the primary telling it to do the expire action (invalidate or destroy) even if the entry does not exist on the primary. We should also consider having a fixed delay of the expiration on the secondary so that if primary expiration is working it will happen before the secondary expiration. If the primary expiration happens first then it will send a message to the secondary cancelling it and expiring the entry there.
Another option is to redefine eviction so that it accommodates to expiration instead of expiration accommodating to eviction. Eviction doing a local destroy on a region that provides consistent storage (both partitioned and replicate regions) is not a good thing. It would be much better for eviction to do a distributed destroy on these regions.