Major Description
Problem: Inconsistent bucket copy when a bucket region is created due to primary bucket event event tracker out of sequence.
Fix: Ensure event tracker is initialized after create region.
When a bucket region is created, it send CreateRegionMessage to all nodes hosting the bucket data. It initiates its event tracker from the first one replied.
In one case, it copies event tracker states from the node with primary bucket, which is processing putAll operation and already applied a few entry operations (so the newer seqNo is recorded for the thread performing the putAll).
However, it gets initial image from another node – which does not have the entry operations yet (as putAll op is not yet being distributed to secondary yet.)
The newly created region would receive all the putAll operations from the primary when the primary distributes the putAll operations to secondary copies. In the node with newly created region, some of the events would not be applied due to hasSeenEvent method call (up to initiated last seqNo for the said thread). This leads to bucket inconsistence among the redundant copies.
Please note this issue would not occur if there is only one redundant copy.