[IGNITE-6355] Calculating cache size during cache stop sporadically fails with ClusterGroupEmptyCheckedException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1
• Fix Version/s: 2.3
• Labels: MakeTeamcityGreenAgain
• Assignee: Ivan Rakov
• Reporter: Ivan Rakov
• Created: 12/Sep/17 13:30
• Updated: 20/Sep/17 09:49
• Resolved: 20/Sep/17 09:49

Description

Example stacktrace:

[16:21:06,343][ERROR][main][root] Test failed.
javax.cache.CacheException: class org.apache.ignite.cluster.ClusterGroupEmptyException: Topology projection is empty.
	at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1327)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1672)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:762)
	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.size(GatewayProtectedCacheProxy.java:508)
	at org.gridgain.grid.internal.processors.cache.database.IgniteDbSnapshotSelfTest.testReuseCacheProxyAfterRestore(IgniteDbSnapshotSelfTest.java:1793)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
	at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
	at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
	at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.cluster.ClusterGroupEmptyException: Topology projection is empty.
	at org.apache.ignite.internal.util.IgniteUtils$6.apply(IgniteUtils.java:823)
	at org.apache.ignite.internal.util.IgniteUtils$6.apply(IgniteUtils.java:821)
	... 14 more
Caused by: class org.apache.ignite.internal.cluster.ClusterGroupEmptyCheckedException: Topology projection is empty.
	at org.apache.ignite.internal.processors.task.GridTaskWorker.getTaskTopology(GridTaskWorker.java:665)
	at org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:500)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:758)
	at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:454)
	at org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:410)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.sizeAsync(GridCacheAdapter.java:3747)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:3704)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.size(IgniteCacheProxyImpl.java:759)
	... 11 more

Data race stems from here (GridCacheAdapter#sizeAsync):

        Group grp = modes.near ? cluster.forCacheNodes(name(), true, true, false) : cluster.forDataNodes(name());

        Collection<ClusterNode> nodes = grp.nodes();

        if (nodes.isEmpty())
            return new GridFinishedFuture<>(0);

        ctx.kernalContext().task().setThreadContext(TC_SUBGRID, nodes);

        return ctx.kernalContext().task().execute(
            new SizeTask(ctx.name(), ctx.affinity().affinityTopologyVersion(), peekModes), null);

Method grp.nodes() returns PredicateCollectionView, which size depends on Ignite state. It can pass nodes.isEmpty() check and become empty later.

Issue Links

• is a child of: IGNITE-6452 Invocation of getAll() through cache proxy during cache restart can throw unexpected CacheException
• links to: GitHub Pull Request #2647