[IGNITE-23231] [FLAKY] Exception "Send with retry timed out" while creating 1000 tables in 10 threads with replication 1

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: 3.0
• Component/s: general, persistence
• Labels: ignite-3

Environment

Cluster of 1 server located on 1 host.

• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Ivan Bessonov
• Reporter: Igor
• Created: 18/Sep/24 17:55
• Updated: 07/Nov/24 12:26
• Resolved: 07/Nov/24 12:26

Description

Steps to reproduce:

Cluster of 1 server located on 1 host.

1. In 10 threads create 50 tables:
   1. Create zone if not exists (same name for all tables) with replication 1.
   2. Create table (all tables have different names).
2. In 1 thread assert tables count
3. In 10 threads: insert 1 row in every table created in step 1
4. In 10 threads: assert tables rows are correct.
5. Repeat steps 1-4 until 1000 tables are created.

 

Expected:
All tables are created.

Actual:
Exception on client side after 850-950 tables:

java.sql.SQLException: Send with retry timed out [retryCount = 5].  at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)  at app//org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)  at app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:53)  at app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:40)  at app//org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3183)  at app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$14(TablesAmountCapacityBaseTest.java:361)  at app//io.qameta.allure.Allure.lambda$step$0(Allure.java:113)  at app//io.qameta.allure.Allure.lambda$step$1(Allure.java:127)  at app//io.qameta.allure.Allure.step(Allure.java:181)  at app//io.qameta.allure.Allure.step(Allure.java:125)  at app//io.qameta.allure.Allure.step(Allure.java:112)  at app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.step(TablesAmountCapacityBaseTest.java:255)  at app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.invokeTasks(TablesAmountCapacityBaseTest.java:356)  at app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.createTablesParallel(TablesAmountCapacityBaseTest.java:130)  at app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.testTablesAmount(TablesAmountCapacityBaseTest.java:91)  at app//org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityTest.create1000EmptyTablesAmountOfColumnsEachAndMakeSimpleQueries(TablesAmountCapacityTest.java:86)  at java.base@17.0.6/java.lang.reflect.Method.invoke(Method.java:568)  at app//io.qameta.allure.junit5.AllureJunit5.interceptTestTemplateMethod(AllureJunit5.java:59)  at java.base@17.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)  at java.base@17.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)  at java.base@17.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)  at java.base@17.0.6/java.lang.Thread.run(Thread.java:833)Caused by: java.util.concurrent.ExecutionException: java.sql.SQLException: Send with retry timed out [retryCount = 5].  at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)  at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)  at org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$invokeTasks$13(TablesAmountCapacityBaseTest.java:363)  at org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49)  ... 19 moreCaused by: java.sql.SQLException: Send with retry timed out [retryCount = 5].  at org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)  at org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:154)  at org.apache.ignite.internal.jdbc.JdbcStatement.executeUpdate(JdbcStatement.java:177)  at org.gridgain.qaaas.sql.generator.TableGeneratorHelper.createTable(TableGeneratorHelper.java:98)  at org.gridgain.ai3tests.tests.amountcapacity.TablesAmountCapacityBaseTest.lambda$createTablesParallel$0(TablesAmountCapacityBaseTest.java:122)  ... 4 more 

Exception on server side:

2024-09-18 04:31:50:281 +0000 [ERROR][%TablesAmountCapacityTest_cluster_0%Raft-Group-Client-18][LeaseUpdater] Lease update invocation failed
java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException: Send with retry timed out [retryCount = 4].
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
    at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:589)
    at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$handleThrowable$41(RaftGroupServiceImpl.java:661)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.TimeoutException: Send with retry timed out [retryCount = 4].
    at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:590)
    ... 7 more 

Server logs: server-logs.zip

Issue Links

• is caused by: IGNITE-23220 Avoid concurrent Catalog writes to Metastorage on the same node
• supercedes: IGNITE-22814 Exception `Max retry limit exceeded: 10` while creating 1000 tables in 10 threads with replication 1