[IGNITE-10878] "Failed to find completed exchange future" error occurs in test with round cluster restart

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8
• Assignee: Sergey Chugunov
• Reporter: Dmitry Sherstobitov
• Created: 10/Jan/19 09:13
• Updated: 23/Jan/19 08:44
• Resolved: 23/Jan/19 08:41

Description

1. Start cluster, create caches with no persistence and load data into it
2. Restart each node in cluster by order (coordinator first)
   Do not wait until topology message occurs 
3. At some moment there is possibility of error (1 out of 20 runs)

This is the case when the topology version has time to be reset

[23:27:17,218][INFO][exchange-worker-#62][GridCacheProcessor] Started cache [name=ENTITY_CONFIG, id=23889694, memoryPolicyName=no-evict, mode=REPLICATED, atomicity=ATOMIC, backups=2147483647]
[23:27:17,222][SEVERE][exchange-worker-#62][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=83bd0a25-4574-4723-9594-b95ddaab19be, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 172.25.1.40], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47503, /127.0.0.1:47503, /172.17.0.1:47503, lab40.gridgain.local/172.25.1.40:47503], discPort=47503, order=1, intOrder=1, lastExchangeTime=1547065626462, loc=true, ver=2.4.13#20181228-sha1:9033812f, isClient=false], topVer=1, nodeId8=83bd0a25, msg=null, type=NODE_JOINED, tstamp=1547065636782], nodeId=83bd0a25, evt=NODE_JOINED]
class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$18.applyx(CacheAffinitySharedManager.java:1798)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$18.applyx(CacheAffinitySharedManager.java:1743)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllRegisteredCacheGroups(CacheAffinitySharedManager.java:1107)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.initCoordinatorCaches(CacheAffinitySharedManager.java:1743)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCoordinatorCaches(GridDhtPartitionsExchangeFuture.java:573)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:679)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2398)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
[23:27:17,222][INFO][exchange-worker-#62][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], resVer=null, err=class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.]
[23:27:17,238][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$18.applyx(CacheAffinitySharedManager.java:1798)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$18.applyx(CacheAffinitySharedManager.java:1743)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllRegisteredCacheGroups(CacheAffinitySharedManager.java:1107)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.initCoordinatorCaches(CacheAffinitySharedManager.java:1743)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCoordinatorCaches(GridDhtPartitionsExchangeFuture.java:573)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:679)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2398)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)
[23:27:17,238][INFO][exchange-worker-#62][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=83bd0a25-4574-4723-9594-b95ddaab19be, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=83bd0a25-4574-4723-9594-b95ddaab19be, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 172.25.1.40], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47503, /127.0.0.1:47503, /172.17.0.1:47503, lab40.gridgain.local/172.25.1.40:47503], discPort=47503, order=1, intOrder=1, lastExchangeTime=1547065626462, loc=true, ver=2.4.13#20181228-sha1:9033812f, isClient=false], done=true], topVer=null, durationFromInit=420]

Issue Links

• links to: GitHub Pull Request #5853
• links to: TC Run