[IGNITE-13132] Countdown latched gets reinitialize to original value(4) when one or more (but not all) node goes down.

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.6
• Component/s: data structures
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Akash Shinde
• Created: 08/Jun/20 09:55
• Updated: 08/Jun/20 09:57

Description

Ignite version:2.6.0

We are using ignite's distributed countdownlatch to make sure that cache loading is completed on all server nodes. We do this to make sure that our kafka consumers starts only after cache loading is complete on all server nodes. This is the basic criteria which needs to be fulfilled before starts actual processing

 We have 4 server nodes and countdownlatch is initialized to 4. We use "cache.loadCache" method to start the cache loading. When each server completes cache loading it reduces the count by 1 using countDown method. So when all the nodes completes cache loading, the count reaches to zero. When this count  reaches to zero we start kafka consumers on all server nodes.

 But we saw weird behavior in prod env. The 3 server nodes were shut down at the same time. But 1 node is still alive. When this happened the count down was reinitialized to original value i.e. 4. But I am not able to reproduce this in dev env.  
 
Note: Partiton loss were happened when three node gone down at same time.