Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-10873

CorruptedTreeException during simultaneous cache put operations

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.7
    • 2.7.5
    • cache, persistence, sql
    • None

    Description

      [2019-01-09 20:47:04,376][ERROR][pool-9-thread-9][GridDhtAtomicCache] <Cache> Unexpected exception during cache update
      org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@780acfb4[ key: ...... ][ GTEST, null, 254, null, null, null, null, 0, null, null, null, null, null, null, null, 0, 0, 0, null, 0, 0, 0, 0, 0, 0, 0, null, 0, 0, null, 0, null, 0, null, 0, null, null, null, 0, 0, 0, 0, 0, 0, null, null, null, null, null, null, null, 0.0, 0, 0.0, 0, 0.0, 0, null, 0, 0, 0, 0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null ]" [50000-197]
      	at org.h2.message.DbException.get(DbException.java:168)
      	at org.h2.message.DbException.convert(DbException.java:307)
      	at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:302)
      	at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.addToIndex(GridH2Table.java:546)
      	at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:479)
      	at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:768)
      	at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1905)
      	at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:404)
      	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2633)
      	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1646)
      	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1621)
      	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1935)
      	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:428)
      	at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2295)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2494)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1951)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1780)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.map(GridNearAtomicSingleUpdateFuture.java:483)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:443)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1153)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:611)
      	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2449)
      	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2426)
      	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1105)
      	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
      	at IndexCorruptionReproducer$1.run(IndexCorruptionReproducer.java:43)
      	at java.util.concurrent.Executors$RunnableAdapter.call$$$capture(Executors.java:511)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java)
      	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
      	at java.util.concurrent.FutureTask.run(FutureTask.java)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@780acfb4[ key:  [i......] ][ GTEST, null, 254, null, null, null, null, 0, null, null, null, null, null, null, null, 0, 0, 0, null, 0, 0, 0, 0, 0, 0, 0, null, 0, 0, null, 0, null, 0, null, 0, null, null, null, 0, 0, 0, 0, 0, 0, null, null, null, null, null, null, null, 0.0, 0, 0.0, 0, 0.0, 0, null, 0, 0, 0, 0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null ]" [50000-197]
      	at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
      	... 36 more
      Caused by: class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@780acfb4[ key:....] ][ GTEST, null, 254, null, null, null, null, 0, null, null, null, null, null, null, null, 0, 0, 0, null, 0, 0, 0, 0, 0, 0, 0, null, 0, 0, null, 0, null, 0, null, 0, null, null, null, 0, 0, 0, 0, 0, 0, null, null, null, null, null, null, null, 0.0, 0, 0.0, 0, 0.0, 0, null, 0, 0, 0, 0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null ]
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2285)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2232)
      	at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:299)
      	... 33 more
      Caused by: java.lang.IllegalStateException: Duplicate row in index.
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:436)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Insert.run0(BPlusTree.java:422)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:344)
      	at org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:276)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$12000(BPlusTree.java:90)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.tryInsert(BPlusTree.java:3583)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.access$7300(BPlusTree.java:3263)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2530)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2511)
      	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2252)
      	... 35 moretion.java:168)
      

      Attachments

        Issue Links

          Activity

            Need to look into on-heap cache. If on-heap cache is disabled, then reproducer passes, otherwise "Duplicate row in index" error occurs.

            agoncharuk Alexey Goncharuk added a comment - Need to look into on-heap cache. If on-heap cache is disabled, then reproducer passes, otherwise "Duplicate row in index" error occurs.

            It seems to me that it is directly related to H2 row cache. If you remove cache usage from H2Tree.createRow() it will work all right. I imagine that maybe it will get the wrong row from the same link due to fast cycling of row addresses in this scenario.

            ilyak Ilya Kasnacheev added a comment - It seems to me that it is directly related to H2 row cache. If you remove cache usage from H2Tree.createRow() it will work all right. I imagine that maybe it will get the wrong row from the same link due to fast cycling of row addresses in this scenario.

            Or maybe we don't remove row from cache as we remove it from tree.

            ilyak Ilya Kasnacheev added a comment - Or maybe we don't remove row from cache as we remove it from tree.

            We should probably avoid using row cache during inserts/updates/removes (via thread-local maybe?)

            ilyak Ilya Kasnacheev added a comment - We should probably avoid using row cache during inserts/updates/removes (via thread-local maybe?)
            ivan.glukos Ivan Rakov added a comment -

            Seems like the problem was in GridQueryProcessor busy lock. Query processor stops before cache processor, and complex operation (that comprises both index and data store updates) may be stopped in the middle due to busy lock acquisition failure.
            I can't reproduce the issue on PR branch regardless of whether SQL onheap cache is enabled.

            ivan.glukos Ivan Rakov added a comment - Seems like the problem was in GridQueryProcessor busy lock. Query processor stops before cache processor, and complex operation (that comprises both index and data store updates) may be stopped in the middle due to busy lock acquisition failure. I can't reproduce the issue on PR branch regardless of whether SQL onheap cache is enabled.
            ignitetcbot Ignite TC Bot added a comment -
            --> Run :: All: No blockers found!

            TeamCity --> Run :: All Results

            ignitetcbot Ignite TC Bot added a comment - --> Run :: All: No blockers found! TeamCity --> Run :: All Results
            ivan.glukos Ivan Rakov added a comment -

            gvvinblade, can you please take a look?

            ivan.glukos Ivan Rakov added a comment - gvvinblade , can you please take a look?

            ivan.glukos, looks OK, at least I don't see any possible issue goes from the change.

            gvvinblade Igor Seliverstov added a comment - ivan.glukos , looks OK, at least I don't see any possible issue goes from the change.
            ivan.glukos Ivan Rakov added a comment -

            Igor, thanks for review! Merged to master.

            ivan.glukos Ivan Rakov added a comment - Igor, thanks for review! Merged to master.
            dpavlov Dmitry Pavlov added a comment -

            Cherry-picked to 2.7.5

            dpavlov Dmitry Pavlov added a comment - Cherry-picked to 2.7.5

            People

              ivan.glukos Ivan Rakov
              pvinokurov Pavel Vinokurov
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m