1)Start 3 data nodes and activate the cluster with cache with 1 backup and PartitionLossPolicy.READ_ONLY_SAFE.
2)Start client and add the data to your cache. Stop the client
3)Stop DN2 and clear it pds and val
4)Start DN2. Rebalance will start.
5)During rebalance stop DN3.
At this moment some partitions from DN2 marked as LOST and cache size will be less than expected.
7) Run resetLostPartitions(caches).
Now all partitions on DN2 marked as OWNING but cache size is still less than expected.
after step 6 do:
7)force rebalance using deactivate/activate methods.
8)wait for completion of rebalance
Now cache size is expected but some partitions from DN2 marked as LOST
Now cache size is OK and all partitions from DN2 marked as OWNING.
However, looks like without force rebalance we have data loss here.