[IGNITE-22517] Partitions stay DEGRADED after node replacement in 3 node cluster

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-beta2
• Component/s: general, networking, persistence
• Labels: ignite-3

Environment

3 nodes cluster (1 CMG node)

• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Igor
• Created: 16/Jun/24 11:39
• Updated: 6 days ago 18:09

Description

This issue is replaced by https://issues.apache.org/jira/browse/IGNITE-22553 from time to time.

Steps to reproduce:

1. Create 3 nodes cluster with 1 CMG node.
2. Create zone with replication equals to amount of nodes (3).
3. Create 10 tables inside the zone.
4. Insert 100 rows in every table.
5. Await all tables*partitions*nodes local state is "HEALTHY"
6. Await all tables*partitions*nodes global state is "AVAILABLE"
7. Kill non CMG node with kill -9.
8. Create new node and attach it to cluster instead of the killed one (update lookup configs of alive nodes to allow them find).
9. Using REST API await physical topology has 3 alive nodes.
10. Using REST API await logical topology has 3 alive nodes.
11. Await all tables*partitions*nodes local state is "HEALTHY".
12. Await all tables*partitions*nodes global state is "AVAILABLE".

Expected:
All partitions become "AVAILABLE".

Actual:
Partitions in different states: "AVAILABLE" and "DEGRADED".

The logs are in attachment.

 

Issue Links

• relates to: IGNITE-22553 Internal Server Error after node replacement in 3 node cluster
• supercedes: IGNITE-22376 RestAPI get logical topology freeze if 1 node was replaced in 3 nodes cluster