[IGNITE-9975] Possible data loss during starting of the nodes with empty pds

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4
• Fix Version/s: 2.8
• Component/s: persistence
• Assignee: Anton Kalashnikov
• Reporter: Andrei Aleksandrov
• Created: 23/Oct/18 11:28
• Updated: 07/Nov/18 11:31
• Resolved: 07/Nov/18 11:27

Description

Case: 

1)Start 3 data nodes and activate the cluster with cache with 1 backup and PartitionLossPolicy.READ_ONLY_SAFE.
2)Start client and add the data to your cache. Stop the client
3)Stop DN2 and clear it pds and val
4)Start DN2. Rebalance will start.
5)During rebalance stop DN3.
6)Start DN3.

At this moment some partitions from DN2 marked as LOST and cache size will be less than expected.

7) Run resetLostPartitions(caches).

Now all partitions on DN2 marked as OWNING but cache size is still less than expected.

Workaround:

after step 6 do:

7)force rebalance using deactivate/activate methods.
8)wait for completion of rebalance

Now cache size is expected but some partitions from DN2 marked as LOST

9)Run resetLostPartitions(caches).

Now cache size is OK and all partitions from DN2 marked as OWNING.

However, looks like without force rebalance we have data loss here.

 

Issue Links

• Dependency: IGNITE-8893 Blinking node in baseline may corrupt own WAL records
• is a clone of: IGNITE-8888 Possible data loss during starting of the nodes with empty pds
• links to: GitHub Pull Request #5214
• links to: TC All
• links to: UpSource