[IGNITE-6538] Ignite Activate/Deactivate Cluster suite: After tests validation improvements test became flaky on TC

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 2.1
• Fix Version/s: 2.7
• Component/s: persistence
• Labels: MakeTeamcityGreenAgain Muted_test
• Assignee: Unassigned
• Reporter: Dmitry Pavlov
• Created: 02/Oct/17 12:21
• Updated: 11/Dec/18 12:41
• Resolved: 09/Aug/18 08:25

Description

https://ci.ignite.apache.org/project.html?projectId=Ignite20Tests&testNameId=-238119157387028099&tab=testDetails&branch_Ignite20Tests=%3Cdefault%3E

[14:36:09]W:		 [org.apache.ignite:ignite-core] [2018-05-22 11:36:09,817][WARN ][exchange-worker-#202059%node1-backUp%][GridDhtPartitionsExchangeFuture] Unable to await partitions release latch within timeout: ClientLatch [coordinator=TcpDiscoveryNode [id=ba7f90d8-3a1f-488a-a8ea-3ae2fdf016fe, addrs=[127.0.0.1, 172.17.0.7], sockAddrs=[/127.0.0.1:47503, d377975c5791/172.17.0.7:47503], discPort=47503, order=1, intOrder=1, lastExchangeTime=1526985998704, loc=false, ver=2.5.0#20180522-sha1:d37470c7, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=1]]]

Fail test:

org.apache.ignite.internal.processors.cache.persistence.standbycluster.IgniteChangeGlobalStateTest#testActivateAfterFailGetLock

Fail test 2:

IgniteChangeGlobalStateTest#testConcurrentActivateFromClientNodeAndServerNode

As result of the test there is page corruption in metastorage:

[org.apache.ignite:ignite-core] java.lang.AssertionError: calculatedOffset=3072, allocated=2048, headerSize=1024
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:346)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:341)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:325)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:142)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:167)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:370)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage$FreeListImpl.<init>(MetaStorage.java:472)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.init(MetaStorage.java:139)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:808)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:901)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[13:46:39]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:748)

Issue Links

• duplicates: IGNITE-9004 Failed to reinitialize local partitions
• is blocked by: IGNITE-8469 Non-heap memory leak for calling cluster activation multi times
• links to: GitHub Pull Request #4044