Blocker Description
Replicated cache sometimes won't sync across nodes properly.
PFA a reproducer code.
All nodes are started at the same time on different machines:
- Ignition.start() // Blocks until node is up
- Only one of the nodes performs next: getOrCreateCache() then putAll()
- All the other nodes block on this before proceeding.
- All of the nodes perform next:
- getOrCreateCache() // Again
- cache.localSize(CachePeekMode.ALL)
All nodes should see filled cache, but sometimes some nodes see empty cache. LocalSize call can be replaced by iterating over cache, but result will be same.
Much more rarely, cluster degradation is possible and one part of cluster see empty cache while another see filled cache. Logs contain no errors at all. It takes about two hours running test in infinite loop to catch this rare error.