[IGNITE-22584] Cluster load balancing when the second node (non-CMG) is killed doesn't work

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta1
• Component/s: persistence, sql
• Labels: ignite-3
• Release Note: root case of a problem is attached
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Evgeny Stanilovsky
• Reporter: Anatoli Karman
• Created: 26/Jun/24 10:32
• Updated: 18/Jul/24 05:32
• Resolved: 18/Jul/24 05:32

Description

Steps to reproduce:

1. Start cluster with 2 nodes running locally.
2. Make connection like this:

try (IgniteClient igniteClient = IgniteClient.builder().retryPolicy(new RetryLimitPolicy()).addresses(thinClientEndpoints.toArray(new String[]{"localhost:10800","localhost:10801"})).build()) {
    try (Session session = igniteClient.sql().createSession()) {
        //code here
    }
} 

3. Create table with replication 2

4. Execute insert 1 row and select from the table.
5. Kill the second node  (non-CMG)
6. Execute select from the table.

Expected:
Cluster works with one node.
Actual:
The exception on select after first node is killed, the select is not executed.

org.opentest4j.AssertionFailedError: Select after node is killed ==> Unexpected exception thrown: org.apache.ignite.sql.SqlException: Unable to request next batch: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.120.6.2:3345
        at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
        at app//org.junit.jupiter.api.AssertDoesNotThrow.createAssertionFailedError(AssertDoesNotThrow.java:84)
        at app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:53)
        at app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:40)
        at app//org.junit.jupiter.api.Assertions.assertDoesNotThrow(Assertions.java:3183)
        at app//org.gridgain.ai3tests.tests.ConnectionAfterNodeIsKilledTest.testThinClientConnectionToMultipleHostAfter1NodeIsKilled(ConnectionAfterNodeIsKilledTest.java:128)
        at java.base@11.0.17/java.lang.reflect.Method.invoke(Method.java:566)
        at java.base@11.0.17/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.17/java.lang.Thread.run(Thread.java:834)
      Caused by: org.apache.ignite.sql.SqlException: IGN-CMN-65535 TraceId:e1244baf-7528-40e5-aa3a-ec8306c4c41a Unable to request next batch: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.120.6.2:3345
        at java.base@11.0.17/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
        at app//org.apache.ignite.internal.util.ExceptionUtils$1.copy(ExceptionUtils.java:789)
        at app//org.apache.ignite.internal.util.ExceptionUtils$ExceptionFactory.createCopy(ExceptionUtils.java:723)
        at app//org.apache.ignite.internal.util.ExceptionUtils.copyExceptionWithCause(ExceptionUtils.java:525)
        at app//org.apache.ignite.internal.util.ExceptionUtils.copyExceptionWithCauseInternal(ExceptionUtils.java:658)
        at app//org.apache.ignite.internal.util.ExceptionUtils.copyExceptionWithCause(ExceptionUtils.java:476)
        at app//org.apache.ignite.internal.client.sql.ClientSql.execute(ClientSql.java:102)
        at app//org.gridgain.ai3tests.tests.teststeps.ThinClientSteps.lambda$executeQuery$0(ThinClientSteps.java:64)
        at app//io.qameta.allure.Allure.lambda$step$1(Allure.java:127)
        at app//io.qameta.allure.Allure.step(Allure.java:181)
        at app//io.qameta.allure.Allure.step(Allure.java:125)
        at app//org.gridgain.ai3tests.tests.teststeps.ThinClientSteps.executeQuery(ThinClientSteps.java:64)
        at app//org.gridgain.ai3tests.tests.TestUtils.selectAll(TestUtils.java:174)
        at app//org.gridgain.ai3tests.tests.ConnectionAfterNodeIsKilledTest.lambda$testThinClientConnectionToMultipleHostAfter1NodeIsKilled$0(ConnectionAfterNodeIsKilledTest.java:129)
        at app//org.junit.jupiter.api.AssertDoesNotThrow.assertDoesNotThrow(AssertDoesNotThrow.java:49)
        ... 8 more
      Caused by: java.util.concurrent.CompletionException: org.apache.ignite.sql.SqlException: IGN-CMN-65535 TraceId:e1244baf-7528-40e5-aa3a-ec8306c4c41a Unable to request next batch: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.120.6.2:3345
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
        at org.apache.ignite.internal.client.TcpClientChannel.processNextMessage(TcpClientChannel.java:426)
        at org.apache.ignite.internal.client.TcpClientChannel.lambda$onMessage$3(TcpClientChannel.java:245)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
      Caused by: org.apache.ignite.sql.SqlException: IGN-CMN-65535 TraceId:e1244baf-7528-40e5-aa3a-ec8306c4c41a Unable to request next batch: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.120.6.2:3345
        at java.base@11.0.17/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
        at app//org.apache.ignite.internal.util.ExceptionUtils$1.copy(ExceptionUtils.java:789)
        at app//org.apache.ignite.internal.util.ExceptionUtils$ExceptionFactory.createCopy(ExceptionUtils.java:723)
        at app//org.apache.ignite.internal.util.ExceptionUtils.copyExceptionWithCause(ExceptionUtils.java:525)
        at app//org.apache.ignite.internal.client.TcpClientChannel.readError(TcpClientChannel.java:510)
        at app//org.apache.ignite.internal.client.TcpClientChannel.processNextMessage(TcpClientChannel.java:404)
        ... 7 more

Issue Links

• Dependent: IGNITE-22768 Sql. Worker node left the cluster before fragment has been sent
• relates to: IGNITE-21663 Cluster load balancing when 1 node is killed doesn't work