Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-7414

Cluster restart may lead to cluster activation error

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Cannot Reproduce
    • 2.3
    • 2.9, 2.8.1
    • general

    Description

      Attempt to execute the following reproducer twice result in an error (please see attached)

          public static void main(String[] args) throws IgniteException {
              Ignite ignite = Ignition.start(createIgniteConfiguration());
      
              ignite.active(true);
      
              CacheConfiguration cacheCfg = new CacheConfiguration("test-cache");
              cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
              cacheCfg.setCacheMode(CacheMode.PARTITIONED);
              cacheCfg.setDataRegionName("inmemory");
              cacheCfg.setIndexedTypes(Integer.class, String.class);
      
              IgniteCache cache = ignite.getOrCreateCache(cacheCfg);
              cache.put(42, "value-42");
      
              ignite.close();
          }
      
          private static IgniteConfiguration createIgniteConfiguration() {
              IgniteConfiguration ignCfg = IgniteConfiguration();
              ...
      
              DataStorageConfiguration dc = new DataStorageConfiguration();
      
              // persistence enabled region
              DataRegionConfiguration persistenceRegion = new DataRegionConfiguration();
              persistenceRegion.setName("persistence");
              persistenceRegion.setPersistenceEnabled(true);
      
              // persistence disabled region
              DataRegionConfiguration inmemoryRegion = new DataRegionConfiguration();
              inmemoryRegion.setName("inmemory");
              inmemoryRegion.setInitialSize(100L * 1024 * 1024);
              inmemoryRegion.setMaxSize(500L * 1024 * 1024);
              inmemoryRegion.setPersistenceEnabled(false);
      
              dc.setDataRegionConfigurations(persistenceRegion, inmemoryRegion);
              ignCfg.setDataStorageConfiguration(dc);
              return ignCfg;
          }
      

      The second execution failed with the exception as follows:

      [2018-01-15 17:12:52,431][ERROR][exchange-worker-#42%test-grid%][GridDhtPartitionsExchangeFuture] Failed to activate node components [nodeId=4ba59aa1-cbc5-4d67-8ca5-b6dd1628c5dc, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
      class org.apache.ignite.IgniteCheckedException: Failed to find cache group descriptor [grpId=623628935]
      	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.getPageMemoryForCacheGroup(GridCacheDatabaseSharedManager.java:1602)
      	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1544)
      	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:570)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:820)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:583)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2279)
      	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
      	at java.lang.Thread.run(Thread.java:748)
      Exception in thread "main" class org.apache.ignite.IgniteException: Failed to activate cluster
      	at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:980)
      	at org.apache.ignite.internal.IgniteKernal.active(IgniteKernal.java:3318)
      	at org.apache.ignite.examples.Reproducer.main(Reproducer.java:33)
      Caused by: class org.apache.ignite.IgniteCheckedException: Failed to activate cluster
      	Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find cache group descriptor [grpId=623628935]
      		at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.getPageMemoryForCacheGroup(GridCacheDatabaseSharedManager.java:1602)
      		at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1544)
      		at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:570)
      		at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:820)
      		at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:583)
      		at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2279)
      		at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
      		at java.lang.Thread.run(Thread.java:748)
      [2018-01-15 17:12:52,643][ERROR][exchange-worker-#42%test-grid%][GridCachePartitionExchangeManager] Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=4ba59aa1-cbc5-4d67-8ca5-b6dd1628c5dc, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 172.25.4.158, 2001:0:9d38:6abd:188f:1f82:53e6:fb61], sockAddrs=[DESKTOP-IMRCA1M/172.25.4.158:47500, /2001:0:9d38:6abd:188f:1f82:53e6:fb61:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1516025562983, loc=true, ver=2.4.0#20180115-sha1:31c60bd6, isClient=false], topVer=1, nodeId8=4ba59aa1, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1516025572191]], crd=TcpDiscoveryNode [id=4ba59aa1-cbc5-4d67-8ca5-b6dd1628c5dc, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 172.25.4.158, 2001:0:9d38:6abd:188f:1f82:53e6:fb61], sockAddrs=[DESKTOP-IMRCA1M/172.25.4.158:47500, /2001:0:9d38:6abd:188f:1f82:53e6:fb61:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1516025562983, loc=true, ver=2.4.0#20180115-sha1:31c60bd6, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=4ba59aa1-cbc5-4d67-8ca5-b6dd1628c5dc, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 172.25.4.158, 2001:0:9d38:6abd:188f:1f82:53e6:fb61], sockAddrs=[DESKTOP-IMRCA1M/172.25.4.158:47500, /2001:0:9d38:6abd:188f:1f82:53e6:fb61:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1516025562983, loc=true, ver=2.4.0#20180115-sha1:31c60bd6, isClient=false], topVer=1, nodeId8=4ba59aa1, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1516025572191]], nodeId=4ba59aa1, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=1909804306], init=false, lastVer=GridCacheVersion [topVer=0, order=1516025562402, nodeOrder=0], partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[ExplicitLockReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]], TxReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]], DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]]]], exchActions=null, affChangeMsg=null, initTs=1516025572201, centralizedAff=false, changeGlobalStateE=class o.a.i.IgniteCheckedException: Failed to find cache group descriptor [grpId=623628935], done=true, state=DONE, evtLatch=0, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException: Cluster state change failed., hash=940092163]]
      class org.apache.ignite.IgniteCheckedException: Cluster state change failed.
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:2414)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:2216)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1039)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:649)
      	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2279)
      	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
      	at java.lang.Thread.run(Thread.java:748)
      

      The issue was initially discussed on user-list: http://apache-ignite-users.70518.x6.nabble.com/Error-of-start-with-multiple-data-regions-td19116.html

      Attachments

        1. Reproducer.java
          4 kB
          Vyacheslav Koptilin

        Issue Links

          Activity

            People

              agoncharuk Alexey Goncharuk
              slava.koptilin Vyacheslav Koptilin
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: