[IGNITE-15068] Ignite giving stale data - when data is inserted/updated and immediately read (from the same client node)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9, 2.9.1
• Component/s: cache
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Tanmay Ambre
• Created: 06/Jul/21 16:57
• Updated: 07/Jul/21 12:32

Description

Hi, 

We have a 18 node (server) cluster for Ignite. We have multiple client nodes that connect to ignite and use the cache api to get and put data. 

One of the client node is a Kafka streams application. This application receives an event and for each event:

1. we lookup the data using the key in ignite. i.e. cache.get(key)
2. we update the data if we find an entry in ignite i.e. cache.put(key, value)
3. we insert the data if we don't find an entry in ignite i.e. cache.put(key, value)

This application processes more than 30 million events per day. In some scenarios we get multiple events for the same key "consecutively" i.e. the time difference between consecutive events is not more  than 10 to 20 milliseconds. We are sure they are processed sequentially as they go to the same Kafka partition. 

What we have observed is sometimes, the step #1 gives us an old copy of the data (not the one which was updated as part of #2). 

 

for e.g. when we get the first event we have following value:

Key = 1, value = {version: 1}

when we get the second event for the same key, we update ignite as:

Key = 1, value = {version: 2} //increment version by 1

when we get the third event for the same key and when we lookup the data in ignite instead of getting {version: 2}, we get {version: 1}

also sometimes, when we get the second event - we don't even find the entry in ignite (i.e. {version:1})

Our caches have the following configuration 

backups = 1

atomicityMode = ATOMIC

writeSynchronizationPolicy = PRIMARY_SYNC

readFromBackup = true (default - we are not setting this)

 

Is there something wrong? Should we set "readFromBackup = false"?

unfortunately it is very difficult to replicate since it happens just 50 to 100 times in a day (i.e. out of 30 million events).