[IGNITE-10183] [Test Failed] IgniteClientReconnectCacheTest.testReconnectOperationInProgress flaky fails on TC.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7
• Fix Version/s: 2.8
• Component/s: cache, mvcc
• Labels: MakeTeamcityGreenAgain failover
• Assignee: Roman Kondakov
• Reporter: Pavel Pereslegin
• Created: 08/Nov/18 11:03
• Updated: 11/Mar/19 06:32
• Resolved: 31/Jan/19 17:35

Description

IgniteClientReconnectCacheTest.testReconnectOperationInProgress fails sometimes in master (see recent failures on TC).

Test fails only for cache with TRANSACTIONAL_SNAPSHOT atomicity mode.

Typical output:

[2018-11-07 23:13:00,579][ERROR][main][root] Test failed.
class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.transactions.TransactionRollbackException: Transaction has been rolled back: 2e10370f661-00000000-0920-4f4e-0000-000000000026
    at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7425)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:261)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:191)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
    at org.apache.ignite.internal.IgniteClientReconnectCacheTest.checkOperationInProgressFails(IgniteClientReconnectCacheTest.java:1421)
    at org.apache.ignite.internal.IgniteClientReconnectCacheTest.testReconnectOperationInProgress(IgniteClientReconnectCacheTest.java:869)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at junit.framework.TestCase.runTest(TestCase.java:176)
    at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2208)
    at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:144)
    at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2124)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.cache.CacheException: class org.apache.ignite.transactions.TransactionRollbackException: Transaction has been rolled back: 2e10370f661-00000000-0920-4f4e-0000-000000000026
    at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1108)
    at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
    at org.apache.ignite.internal.IgniteClientReconnectCacheTest$10.apply(IgniteClientReconnectCacheTest.java:834)
    at org.apache.ignite.internal.IgniteClientReconnectCacheTest$10.apply(IgniteClientReconnectCacheTest.java:832)
    at org.apache.ignite.internal.IgniteClientReconnectCacheTest$25.call(IgniteClientReconnectCacheTest.java:1406)
    at org.apache.ignite.testframework.GridTestUtils.lambda$runAsync$2(GridTestUtils.java:1003)
    at org.apache.ignite.testframework.GridTestUtils$7.call(GridTestUtils.java:1299)
    at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84)
Caused by: class org.apache.ignite.transactions.TransactionRollbackException: Transaction has been rolled back: 2e10370f661-00000000-0920-4f4e-0000-000000000026
    at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:922)
    at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:920)
    ... 10 more
Caused by: class org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException: Transaction has been rolled back: 2e10370f661-00000000-0920-4f4e-0000-000000000026
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4282)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2503)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2484)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2461)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
    ... 7 more
Caused by: class org.apache.ignite.IgniteCheckedException: Cannot serialize transaction due to write conflict (transaction is marked for rollback)
    at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7425)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:261)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:172)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture$2.apply(GridDhtTxAbstractEnlistFuture.java:515)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture$2.apply(GridDhtTxAbstractEnlistFuture.java:512)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:395)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:507)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:486)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
    at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$MvccUpdateLockListener.apply(GridCacheMapEntry.java:5445)
    at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$MvccUpdateLockListener.apply(GridCacheMapEntry.java:5303)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:395)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:507)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:486)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:452)
    at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl$LockFuture.run(MvccProcessorImpl.java:1962)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.internal.processors.query.IgniteSQLException: Cannot serialize transaction due to write conflict (transaction is marked for rollback)
    at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.serializationError(GridCacheMapEntry.java:6740)
    at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.access$700(GridCacheMapEntry.java:127)
    ... 12 more

Test invokes cache put/get from client node and at the same time forces client to disconnect.
With ATOMIC and TRANSACTIONAL atomicity mode when client disconnects it cancels all futures with exception IgniteClientDisconnectedCheckedException (for example, see GridCacheMvccManager#onDisconnected).
It seems that for the TRANSACTIONAL_SNAPSHOT mode, similar logic should be implemented.

Issue Links

• links to: GitHub Pull Request #5797
• links to: TC run