The issue is very clear now.
When log replay is in place, postOpen() hook is getting called before any replay. In case of Visibility , we use this hook to read the existing labels and initialize the cache. So here what happens is the cache is getting no items at all. Also we calculate the next ordinal no# for adding label in this postOpen(). It comes as 1 again. While doing the 2 new entries write, it gets ordinals as 1 and 2 replacing some old labels !!!
That is why the test fails with number of labels after adding 2 new labels (already 6 in place) as 6 only.
Similar issue is there with AccessController also in which the existing data is cached in postOpen(). If some unflushed acls are in place, then those will be lost to get added into the cache (up on RS restart)..
I am trying with possible solutions.
Can we think of a hook with replay like postReplay() or so?