[IGNITE-21356] ItSqlLogicTest sql(types/decimal/test_decimal_ops.test) failed with IgniteOutOfMemoryException

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: MakeTeamcityGreenAgain ignite-3
• Assignee: Unassigned
• Reporter: Alexander Lapin
• Created: 26/Jan/24 07:13
• Updated: 26/Jan/24 07:26

Description

[2024-01-25T10:28:50,418][INFO ][%sqllogic1%checkpoint-thread-4][Checkpointer] Skipping checkpoint (no pages were modified) [checkpointBeforeWriteLockTime=0ms, checkpointWriteLockWait=0ms, checkpointListenersExecuteTime=0ms, checkpointWriteLockHoldTime=0ms, reason='timeout']------- Stderr: -------[2024-01-25T10:20:17,371][WARN ][%sqllogic0%metastorage-watch-executor-0][WatchProcessor] Watch event processing has been too long [duration=236, keys=[catalog.update.1342, catalog.version]][2024-01-25T10:20:17,374][WARN ][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event processing has been too long [duration=236, keys=[catalog.update.1342, catalog.version]][2024-01-25T10:20:18,268][WARN ][%sqllogic0%metastorage-watch-executor-1][WatchProcessor] Watch event processing has been too long [duration=238, keys=[catalog.update.1343, catalog.version]][2024-01-25T10:20:18,269][WARN ][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event processing has been too long [duration=236, keys=[catalog.update.1343, catalog.version]][2024-01-25T10:20:19,117][WARN ][%sqllogic0%tableManager-io-7][CatalogManagerImpl] Failed to apply catalog update.java.util.concurrent.CompletionException: org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException: IGN-CMN-65535 TraceId:061dbe67-7b05-4773-957a-d568db1c5897 Out of memory in data region [name=default, size=256.0 MiB, persistence=true] Try the following:  ^-- Increase maximum off-heap memory size (PersistentPageMemoryDataRegionConfigurationSchema.size)  ^-- Enable eviction or expiration policies  at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]  at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:683) ~[?:?]  at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) ~[?:?]  at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094) ~[?:?]  at org.apache.ignite.internal.storage.util.MvPartitionStorages.create(MvPartitionStorages.java:120) ~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?]  at org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.lambda$createMvPartition$5(AbstractPageMemoryTableStorage.java:163) ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]

...

Caused by: org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException: Failed to find a page for eviction (no pages to replace) [segmentCapacity=6805, loaded=2709, dirtyPagesSoftThreshold=2031, dirtyPagesHardThreshold=2438, dirtyPages=2709, pinned=0]

https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7799962?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true&expandBuildChangesSection=true

 

There's also suspicious

  Caused by: java.lang.AssertionError: FullPageId [pageId=000100000000000c, effectivePageId=000000000000000c, groupId=770]
    at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:758) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:641) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:613) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:280) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:296) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.pagememory.freelist.PagesList.flushBucketsCache(PagesList.java:387) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.pagememory.freelist.PagesList.saveMetadata(PagesList.java:332) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.storage.pagememory.mv.RowVersionFreeList.saveMetadata(RowVersionFreeList.java:185) ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$syncMetadataOnCheckpoint$13(PersistentPageMemoryMvPartitionStorage.java:345) ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
    at org.apache.ignite.internal.pagememory.persistence.checkpoint.AwaitTasksCompletionExecutor.lambda$execute$1(AwaitTasksCompletionExecutor.java:59) ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    ... 1 more