[IGNITE-22729] (flaky) `Failed to get the primary replica` after 400 tables are created

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0
• Component/s: general, persistence
• Labels: ignite-3

Environment

3 nodes (1 CMG) distributed on 3 hosts with "-Xms4096m", "-Xmx4096m"

• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Igor
• Created: 14/Jul/24 19:47
• Updated: 04/Sep/24 16:22

Description

This issue sometimes is replaced by https://issues.apache.org/jira/browse/IGNITE-22727

Steps to reproduce:

1. Multinode cluster (3 nodes) with arguments 
   "-Xms4096m", "-Xmx4096m"
2. Create 50 tables (all tables in single zone).
3. Insert 1 row into tables created in the previous step.
4. Assert select count from each table equals 1.
5. Repeat steps 2 and 4 until 1000 tables are created.

Expected:
1000 tables are created.

Actual:
After 400 tables created, 1 row inserted in each table, while executing select count from tables from 350 to 400 (step 4), the exception is thrown:

 

java.sql.SQLException: Failed to get the primary replica [tablePartitionId=27_part_0]  at org.apache.ignite.internal.jdbc.proto.IgniteQueryErrorCode.createJdbcSqlException(IgniteQueryErrorCode.java:57)  at org.apache.ignite.internal.jdbc.JdbcStatement.execute0(JdbcStatement.java:154)  at org.apache.ignite.internal.jdbc.JdbcStatement.executeQuery(JdbcStatement.java:111)  at org.gridgain.ai3tests.tests.teststeps.JdbcSteps.executeQuery(JdbcSteps.java:91)  at org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.tryGetActualResult(ClusterFailoverTestBase.java:342)  at org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.lambda$getActualResult$7(ClusterFailoverTestBase.java:321)  at org.gridgain.ai3tests.core.utils.RetryUtils.retryOnAllowedException(RetryUtils.java:61)  at org.gridgain.ai3tests.core.utils.RetryUtils.retryOnAllowedException(RetryUtils.java:36)  at org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.getActualResult(ClusterFailoverTestBase.java:320)  at org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.assertDataIsFilledWithoutErrors(ClusterFailoverTestBase.java:178)  at org.gridgain.ai3tests.tests.failover.ClusterFailover2NodesTest.singleKillAndCheckOtherNodeWorks(ClusterFailover2NodesTest.java:127)  at java.base/java.lang.reflect.Method.invoke(Method.java:566)  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)  at java.base/java.lang.Thread.run(Thread.java:834) 

The logs: servers_logs.zip