[IGNITE-12854] Node failed under load when ATOMIC and TRANSACTIONAL caches created in the same cache group

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.9
• Labels: IEP-80
• Ignite Flags: Release Notes Required
• Assignee: Unassigned
• Reporter: Aleksey Plekhanov
• Created: 02/Apr/20 09:02
• Updated: 04/Jul/22 18:09
• Resolved: 02/Apr/20 21:15

Description

Node failed with assertion error under concurrent puts to ATOMIC and TRANSACTIONAL caches belonging to the same cache group:

java.lang.AssertionError: LWM after HWM: lwm=262, hwm=261
    at org.apache.ignite.internal.processors.cache.PartitionUpdateCounterTrackingImpl.reserve(PartitionUpdateCounterTrackingImpl.java:262)
    at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.getAndIncrementUpdateCounter(IgniteCacheOffheapManagerImpl.java:1601)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getAndIncrementUpdateCounter(GridCacheOffheapManager.java:2162)
    at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.getAndIncrementUpdateCounter(GridDhtLocalPartition.java:1087)
    at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.calculatePartitionUpdateCounters(IgniteTxLocalAdapter.java:520)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1343)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.mapIfLocked(GridDhtTxPrepareFuture.java:709)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1104)
    at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:411)
    at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:577)
    at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
    at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:183)
    at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
    at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
    at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
    at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
    at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
    at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)

Reproducer:

 

public void testPartCounters() throws Exception {
    final AtomicBoolean end = new AtomicBoolean();

    final IgniteEx srv = startGrids(3);

    srv.cluster().active(true);

    IgniteInternalFuture<?> loadFut = runMultiThreadedAsync(() -> {
        while (!end.get() && !fail) {
            List<IgniteCache<Object, Object>> caches = F.asList(
                srv.getOrCreateCache(new CacheConfiguration<>(CACHE_NAME_ATOMIC)
                    .setAtomicityMode(CacheAtomicityMode.ATOMIC)
                    .setGroupName(GROUP_NAME)
                ),
                srv.getOrCreateCache(new CacheConfiguration<>(CACHE_NAME_TX)
                    .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)
                    .setGroupName(GROUP_NAME)
                )
            );

            for (IgniteCache<Object, Object> cache : caches) {
                for (int i = 0; i < ENTRIES; i++)
                    cache.put(i, new byte[1024]);
            }
        }
    }, WORKLOAD_THREADS_CNT, "async_runner");

    try {
        // Let's wait some time.
        loadFut.get(10, TimeUnit.SECONDS);
    }
    catch (Exception e) {
        assertFalse("Failure handler was called. See log above.", fail);

        assertTrue(X.hasCause(e, IgniteFutureTimeoutCheckedException.class));
    }
    finally {
        end.set(true);
    }

    assertFalse("Failure handler was called. See log above.", fail);
}

 

Issue Links

• relates to: IGNITE-12622 Forbid mixed cache groups with both atomic and transactional caches