[IGNITE-22730] (flaky) terminate called after throwing an instance of 'std::bad_alloc'

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0
• Component/s: general, persistence
• Labels: ignite-3

Environment

3 nodes (1 CMG) placed on 1 host with "-Xms4096m", "-Xmx4096m"

or 

1 node (1 CMG) placed on 1 host with "-Xms4096m", "-Xmx4096m"

• Assignee: Unassigned
• Reporter: Igor
• Created: 15/Jul/24 00:45
• Updated: 04/Sep/24 16:22

Description

Steps to reproduce:

1. Multinode cluster (3 nodes) with arguments or 1 node cluster running locally
   "-Xms4096m", "-Xmx4096m"
2. Create 50 tables (all tables in single zone).
3. Insert 1 row into tables created in the previous step.
4. Assert select count from each table equals 1.
5. Repeat steps 2 and 4 until 1000 tables are created.

Expected:
1000 tables are created.

Actual:
After 900 tables created, 1 row inserted in each table, while executing select count from tables from 850 to 900 (step 4), the exception is thrown:

 

java.sql.SQLException: java.util.concurrent.ExecutionException: org.apache.ignite.client.IgniteClientConnectionException: IGN-CLIENT-1 TraceId:f330b416-4914-4d4d-9773-e6cde34379a4 Channel is closed  at org.apache.ignite.internal.jdbc.JdbcStatement.toSqlException(JdbcStatement.java:776)  at org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:148)  at org.apache.ignite.internal.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:111)  at org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.assertTableContains1RowAndColumnsValuesAreCorrect(TablesAmountCapacityBaseTest.java:123)  at org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.assertTablesAreValid(TablesAmountCapacityBaseTest.java:241)  at org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityTest.create1000EmptyTablesAmountOfColumnsEachAndMakeSimpleQueries(TablesAmountCapacityTest.java:96)  at java.base/java.lang.reflect.Method.invoke(Method.java:568)  at io.qameta.allure.junit5.AllureJunit5.interceptTestTemplateMethod(AllureJunit5.java:59)  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  at java.base/java.lang.Thread.run(Thread.java:833)Caused by: java.util.concurrent.ExecutionException: org.apache.ignite.client.IgniteClientConnectionException: IGN-CLIENT-1 TraceId:f330b416-4914-4d4d-9773-e6cde34379a4 Channel is closed  at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)  at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)  at org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:144)  ... 10 moreCaused by: org.apache.ignite.client.IgniteClientConnectionException: IGN-CLIENT-1 TraceId:f330b416-4914-4d4d-9773-e6cde34379a4 Channel is closed  at app//org.apache.ignite.internal.client.TcpClientChannel.close(TcpClientChannel.java:229)  at app//org.apache.ignite.internal.client.TcpClientChannel.onDisconnected(TcpClientChannel.java:261)  at app//org.apache.ignite.internal.client.io.netty.NettyClientConnection.onDisconnected(NettyClientConnection.java:120)  at app//org.apache.ignite.internal.client.io.netty.NettyClientMessageHandler.channelInactive(NettyClientMessageHandler.java:39)  at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)  at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)  at app//io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)  at app//io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:412)  at app//io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:377)  at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)  at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)  at app//io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)  at app//io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1402)  at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)  at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)  at app//io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:900)  at app//io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:811)  at app//io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)  at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)  at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)  at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)  at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)  at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)  at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)  ... 1 more 

server_log_without_hprof.zip

In stderr.log:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)
 

The error in the stderr.log file and hrof files appear not always, most of the times the server just silently dies. The probability of creation hprof is higher in 1 node mode.

Issue Links

• is caused by: IGNITE-22757 Excessive memory usage in schema-related code in SQL