[IGNITE-11460] MVCC: Possible race on coordinator changing on client reconnection.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8
• Component/s: mvcc
• Labels: MakeTeamcityGreenAgain
• Assignee: Nikita Amelchev Assign to me
• Reporter: Nikita Amelchev
• Created: 01/Mar/19 09:40
• Updated: 09/Apr/19 13:28
• Resolved: 09/Apr/19 13:23

Description

I found that the wrong coordinator can be set in case of client reconnect:

assert newCrd.topologyVersion().compareTo(curCrd.topologyVersion()) > 0;

java.lang.AssertionError
    at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl.onCoordinatorChanged(MvccProcessorImpl.java:541)
    at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl.onLocalJoin(MvccProcessorImpl.java:416)
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:851)
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:601)
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2681)
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2719)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
    at java.lang.Thread.run(Thread.java:748)

I have attached reproducer in PR.

The main reason is that coordinator can be changed from discovery event thread when the client already disconnect (disconnection processed in notifier thread and change coordinator on onDisconnected method).
Coordinator can be changed in cases:
1. notifier disco thread: onDisconnected method
2. event disco thread: onDiscovery listener.
and events can be processed with some delay and override coordinator that set in notifier thread.

Issue Links

• is related to: IGNITE-8178 ZookeeperDiscoverySpiTest#testReconnectServersRestart* tests fail on TC
• is related to: IGNITE-10693 MVCC TX: Random server restart tests became failed
• is related to: IGNITE-11624 Discovery SPI: The client can handle events from the previous cluster after reconnect.
• links to: GitHub Pull Request #6211
• links to: TC tests
• links to: Upsource