[IGNITE-12014] Getting affinity for topology version earlier than affinity is calculated for system cache

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.7.5
• Assignee: Unassigned
• Reporter: Mikhail Petrov
• Created: 25/Jul/19 11:47
• Updated: 06/Feb/20 06:18
• Resolved: 06/Feb/20 06:18

Description

The following exception was being caught occasionally on a big cluster (128 nodes) after it's activation and concurrent Ignite#reentrantLock() method call from different nodes. (On 16 nodes cluster this exception was never detected with no code change).

It's attached a presumptive reproducer of that problem which stable fails with the specified exception.

java.lang.IllegalStateException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=cf397493-7528-46dc-bc5a-444f9d500001, consistentId=127.0.0.1:47501, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1564050248387, loc=true, ver=2.8.0#20190725-sha1:00000000, isClient=false], grp=default-volatile-ds-group, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], lastAffChangeTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], head=AffinityTopologyVersion [topVer=2, minorTopVer=2], history=[AffinityTopologyVersion [topVer=2, minorTopVer=2]]]
	at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:802)
	at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:749)
	at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:657)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodesByPartition(GridCacheAffinityManager.java:227)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByPartition(GridCacheAffinityManager.java:273)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByKey(GridCacheAffinityManager.java:264)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByKey(GridCacheAffinityManager.java:288)
	at org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.entryExx(GridDhtColocatedCache.java:161)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.entryEx(GridNearTxLocal.java:4470)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.enlistRead(GridNearTxLocal.java:2709)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.getAllAsync(GridNearTxLocal.java:2188)
	at org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache$4.op(GridDhtColocatedCache.java:204)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOp.op(GridCacheAdapter.java:5644)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOp(GridCacheAdapter.java:4561)
	at org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.getAsync(GridDhtColocatedCache.java:202)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4842)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.repairableGet(GridCacheAdapter.java:4808)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1480)
	at org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:396)
	at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:561)
	at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:556)
	at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.retryTopologySafe(DataStructuresProcessor.java:1664)
	at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:556)
	at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.reentrantLock(DataStructuresProcessor.java:1361)
	at org.apache.ignite.internal.IgniteKernal.reentrantLock(IgniteKernal.java:4065)
	at org.apache.ignite.JcacheExchangeAwaitTest.testJcacheExchangeAwait(JcacheExchangeAwaitTest.java:81)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.apache.ignite.testframework.junits.GridAbstractTest$6.run(GridAbstractTest.java:2069)
	at java.lang.Thread.run(Thread.java:748)

Issue Links

• duplicates: IGNITE-11465 Multiple client leave/join events may wipe affinity assignment history and cause transactions fail
• relates to: IGNITE-8098 Getting affinity for topology version earlier than affinity is calculated because of data race
• relates to: IGNITE-7366 Affinity assignment exception in service processor during multiple nodes join
• relates to: IGNITE-10249 TcpDiscoveryMultiThreadedTest#testCustomEventNodeRestart: Getting affinity for topology version earlier than affinity is calculated
• relates to: IGNITE-10389 Getting affinity for topology version earlier than affinity is calculated
• relates to: IGNITE-1171 Getting affinity for topology version earlier than affinity is calculated