[IGNITE-8178] ZookeeperDiscoverySpiTest#testReconnectServersRestart* tests fail on TC

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8
• Component/s: zookeeper
• Labels: MakeTeamcityGreenAgain Muted_test
• Assignee: Nikita Amelchev
• Reporter: Sergey Chugunov
• Created: 09/Apr/18 11:15
• Updated: 04/Mar/19 15:48
• Resolved: 04/Mar/19 15:48

Description

The following stack trace appears in logs:

class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1698)
    at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1007)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1977)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1720)
    at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1148)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:646)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:882)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:845)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:833)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:799)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGrids(GridAbstractTest.java:683)
    at org.apache.ignite.testframework.junits.GridAbstractTest.startGridsMultiThreaded(GridAbstractTest.java:710)
    at org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.startGridsMultiThreaded(GridCommonAbstractTest.java:507)
    at org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.startGridsMultiThreaded(GridCommonAbstractTest.java:497)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoverySpiTest.reconnectServersRestart(ZookeeperDiscoverySpiTest.java:3513)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoverySpiTest.testReconnectServersRestart_1(ZookeeperDiscoverySpiTest.java:3498)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at junit.framework.TestCase.runTest(TestCase.java:176)
    at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2080)
    at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:140)
    at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1995)
    at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: ZookeeperDiscoverySpi [zkRootPath=/apacheIgnite, zkConnectionString=127.0.0.1:46237,127.0.0.1:38364,127.0.0.1:45674, joinTimeout=0, sesTimeout=10000, clientReconnectDisabled=false, internalLsnr=null]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:905)
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1693)
    ... 24 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to initialize Zookeeper nodes
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.initZkNodes(ZookeeperDiscoveryImpl.java:827)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.startJoin(ZookeeperDiscoveryImpl.java:957)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.joinTopology(ZookeeperDiscoveryImpl.java:775)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.startJoinAndWait(ZookeeperDiscoveryImpl.java:693)
    at org.apache.ignite.spi.discovery.zk.ZookeeperDiscoverySpi.spiStart(ZookeeperDiscoverySpi.java:471)
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
    ... 26 more
Caused by: org.apache.ignite.spi.discovery.zk.internal.ZookeeperClientFailedException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /apacheIgnite
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient.onZookeeperError(ZookeeperClient.java:758)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient.exists(ZookeeperClient.java:276)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.initZkNodes(ZookeeperDiscoveryImpl.java:789)
    ... 31 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /apacheIgnite
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
    at org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient.exists(ZookeeperClient.java:273)
    ... 32 more

Isn't easily reproducible locally, comprehensive investigation is needed for what's going on on TC.

Issue Links

• relates to: IGNITE-10777 Cleanup remainders of JUnit4TestAdapter and other JUnit 3 API involving test suites
• relates to: IGNITE-11460 MVCC: Possible race on coordinator changing on client reconnection.
• links to: GitHub Pull Request #5755
• links to: GitHub Pull Request #5755
• links to: TC tests
• links to: Upsource