[IGNITE-8184] ZookeeperDiscoverySpiTest#testTopologyChangeMultithreaded_RestartZk* tests fail on TC

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7
• Component/s: zookeeper
• Labels: MakeTeamcityGreenAgain
• Assignee: Vitaliy Biryukov
• Reporter: Sergey Chugunov
• Created: 09/Apr/18 12:32
• Updated: 04/Jul/18 17:48
• Resolved: 04/Jul/18 17:48

Description

Tests fail on TC but pass locally.

There are some errors in logs like this:

class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1698)
    at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1007)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1977)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1720)
    at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1148)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:646)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:882)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:845)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:833)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:799)
    at org.apache.ignite.testframework.junits.GridAbstractTest$3.call(GridAbstractTest.java:742)
    at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:86)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: ZookeeperDiscoverySpi [zkRootPath=/apacheIgnite, zkConnectionString=127.0.0.1:45822,127.0.0.1:46661,127.0.0.1:43724, joinTimeout=0, sesTimeout=30000, clientReconnectDisabled=false, internalLsnr=null]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:905)
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1693)
    ... 11 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to initialize Zookeeper nodes
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.initZkNodes(ZookeeperDiscoveryImpl.java:827)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.startJoin(ZookeeperDiscoveryImpl.java:957)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.joinTopology(ZookeeperDiscoveryImpl.java:775)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.startJoinAndWait(ZookeeperDiscoveryImpl.java:693)
    at org.apache.ignite.spi.discovery.zk.ZookeeperDiscoverySpi.spiStart(ZookeeperDiscoverySpi.java:471)
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
    ... 13 more
Caused by: org.apache.ignite.spi.discovery.zk.internal.ZookeeperClientFailedException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /apacheIgnite
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient.onZookeeperError(ZookeeperClient.java:808)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient.exists(ZookeeperClient.java:276)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.initZkNodes(ZookeeperDiscoveryImpl.java:789)
    ... 18 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /apacheIgnite
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient.exists(ZookeeperClient.java:273)
    ... 19 more

Also there is some signs that test may hang:

org.apache.ignite.IgniteException: Timeout of waiting for topology map update [igniteInstanceName=internal.ZookeeperDiscoverySpiTest3, cache=default, cacheId=1544803905, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=1], p=1, readVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], locNode=ZookeeperClusterNode [id=a3c3db5c-d2a6-40b4-8f23-dbb08e400003, addrs=[172.17.0.7], order=4, loc=true, client=false]]
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoverySpiTest.topologyChangeWithRestarts(ZookeeperDiscoverySpiTest.java:1880)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoverySpiTest.testTopologyChangeMultithreaded_RestartZk(ZookeeperDiscoverySpiTest.java:1832)

Investigation is needed.

Issue Links

• requires: IGNITE-8182 ZookeeperDiscoverySpiTest#testRandomTopologyChanges_RestartZk fails on TC
• links to: GitHub Pull Request #4179