[IGNITE-8640] If first createCache fail - Ignite is freezing on next createCache

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6
• Fix Version/s: 2.8
• Component/s: cache
• Assignee: Denis Garus
• Reporter: Nikolay Izhikov
• Created: 29/May/18 13:42
• Updated: 29/Nov/18 08:31
• Resolved: 29/Nov/18 08:27

Description

If first createCache operation fails on some condition inside GridCacheProcessor#validate then second createCache will freeze forever.

Reproducer:

package org.apache.ignite.internal.processors.cache;

import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.internal.IgniteEx;
import org.apache.ignite.testframework.GridTestUtils;
import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;

public class CreateCacheFreezeTest extends GridCommonAbstractTest {
    public void testCreateEncryptedNotPersistedCacheFail() throws Exception {
        IgniteEx ignite = startGrid(0);
        
        GridTestUtils.assertThrowsWithCause(() -> {
            CacheConfiguration<Long, String> cc = new CacheConfiguration<>("failed");

            cc.setEvictionPolicy(new FifoEvictionPolicy());
            cc.setOnheapCacheEnabled(false);

            ignite.createCache(cc);

            return 0;
        }, IgniteCheckedException.class);

        IgniteCache<Long, String> cache = ignite.createCache(new CacheConfiguration<>("default"));

        assertNotNull(cache);
    }
}

Log message:

[2018-05-29 16:38:11,958][ERROR][exchange-worker-#38%cache.CreateCacheFreezeTest0%][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=DynamicCacheChangeBatch [id=67cce1ca361-993dd9c2-f4fe-443b-af43-27e06424e1b0, reqs=[DynamicCacheChangeRequest [cacheName=failed, hasCfg=true, nodeId=a525b74c-aec5-4c62-855a-ff96ef300000, clientStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=[failed], stopCaches=null, startGrps=[failed], stopGrps=[], resetParts=null, stateChangeRequest=null], startCaches=false], affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=a525b74c-aec5-4c62-855a-ff96ef300000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1527601090538, loc=true, ver=2.5.0#19700101-sha1:00000000, isClient=false], topVer=1, nodeId8=a525b74c, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1527601091938]], nodeId=a525b74c, evt=DISCOVERY_CUSTOM_EVT]
java.lang.AssertionError: stopping=false, groupName=null, caches=[]
	at org.apache.ignite.internal.processors.cache.CacheGroupContext.singleCacheContext(CacheGroupContext.java:375)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:197)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.getOrCreatePartition(GridDhtPartitionTopologyImpl.java:828)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.initPartitions(GridDhtPartitionTopologyImpl.java:369)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.java:544)
	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1190)
	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:722)
	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452)
	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
	at java.lang.Thread.run(Thread.java:748)

Issue Links

• is duplicated by: IGNITE-9439 Grid hangs after cache start failure.
• is related to: IGNITE-9347 Attempt to start a dynamic cache with invalid configuration leads to exchange worker death
• is related to: IGNITE-9907 Wrong index field name makes the whole cluster to fail
• is related to: IGNITE-1094 Ignite.createCache(CacheConfiguration) hangs if some exception occurs during cache initialization
• relates to: IGNITE-9955 Fail of cache creation on cluster activation
• links to: Apache Ignite Developers topic
• links to: GitHub Pull Request #4480
• links to: TC RunAll
• links to: UP