[IGNITE-6452] Invocation of getAll() through cache proxy during cache restart can throw unexpected CacheException

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1
• Labels: MakeTeamcityGreenAgain
• Assignee: Ivan Rakov
• Reporter: Ivan Rakov
• Created: 20/Sep/17 09:47
• Updated: 28/Sep/17 15:27

Description

Instead of expected IgniteCacheRestartingException, load test shows the following exception sometimes:

javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to find message handler for message: GridNearGetRequest [futId=6fc73459e51-84b93e3c-47e1-433c-8a91-0700f131c617, miniId=27d73459e51-84b93e3c-47e1-433c-8a91-0700f131c617, ver=null, keyMap=null, flags=1, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=32], subjId=080177d4-b78e-4f6f-a386-77be88300000, taskNameHash=0, createTtl=-1, accessTtl=-1]

	at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1285)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1648)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.getAll(IgniteCacheProxyImpl.java:873)
	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.getAll(GatewayProtectedCacheProxy.java:718)
	at org.gridgain.grid.internal.processors.cache.database.IgniteDbSnapshotSelfTest$15.apply(IgniteDbSnapshotSelfTest.java:1911)
	at org.gridgain.grid.internal.processors.cache.database.IgniteDbSnapshotSelfTest$15.apply(IgniteDbSnapshotSelfTest.java:1904)
	at org.gridgain.grid.internal.processors.cache.database.IgniteDbSnapshotSelfTest.testReuseCacheProxyAfterRestore(IgniteDbSnapshotSelfTest.java:1796)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
	at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
	at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
	at java.lang.Thread.run(Thread.java:745)

Also, there is an issue with the case when we start cache operation during the exchange which will start cache.

Caused by: java.lang.IllegalStateException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=21b13959-8579-47dd-86c8-b16ec8500000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1506550922327, loc=true, ver=2.3.0#19700101-sha1:00000000, isClient=false], grp=cache1, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=3], head=AffinityTopologyVersion [topVer=3, minorTopVer=4], history=[AffinityTopologyVersion [topVer=3, minorTopVer=4]]]
	at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:519)
	at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:419)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodesByPartition(GridCacheAffinityManager.java:220)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByPartition(GridCacheAffinityManager.java:256)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByKey(GridCacheAffinityManager.java:247)
	at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByKey(GridCacheAffinityManager.java:271)
	at org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.entryExx(GridDhtColocatedCache.java:156)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.entryEx(GridNearTxLocal.java:3699)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.enlistWriteEntry(GridNearTxLocal.java:1136)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.enlistWrite(GridNearTxLocal.java:855)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.putAsync0(GridNearTxLocal.java:533)
	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.putAsync(GridNearTxLocal.java:385)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2359)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2357)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4057)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2357)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2338)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2315)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1005)
	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
	at org.gridgain.grid.internal.processors.cache.database.SnapshotRestartCacheProxyTest$2.run(SnapshotRestartCacheProxyTest.java:216)
	at org.apache.ignite.testframework.GridTestUtils$6.run(GridTestUtils.java:891)
	at org.apache.ignite.testframework.GridTestUtils$9.call(GridTestUtils.java:1236)
	at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:86)

It happens because we complete restart future before exchange is done.

Issue Links

• is a parent of: IGNITE-6355 Calculating cache size during cache stop sporadically fails with ClusterGroupEmptyCheckedException