Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.15.0
Description
There is sequence of cascading exceptions that occur in the VMs and we need a more detailed investigation: Possible culprit may be the bind address in use exeception:
[*http://files.apachegeode-ci.info/builds/apache-develop-mass-test-run/1.15.0-build.0513/test-results/distributedTest/1632566050/*]
org.apache.geode.management.internal.configuration.ClusterConfigLocatorRestartDUnitTest > serverRestartsAfterLocatorReconnects FAILED org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.management.internal.configuration.ClusterConfigLocatorRestartDUnitTest$$Lambda$163/1450009752.run in VM 0 running on Host heavy-lifter-2c03c48d-8a0c-58ae-bad6-31f64bb5400a.c.apachegeode-ci.internal with 5 VMs at org.apache.geode.test.dunit.VM.executeMethodOnObject(VM.java:631) at org.apache.geode.test.dunit.VM.invoke(VM.java:448) at org.apache.geode.test.junit.rules.VMProvider.invoke(VMProvider.java:94) at org.apache.geode.management.internal.configuration.ClusterConfigLocatorRestartDUnitTest.waitForLocatorToReconnect(ClusterConfigLocatorRestartDUnitTest.java:225) at org.apache.geode.management.internal.configuration.ClusterConfigLocatorRestartDUnitTest.serverRestartsAfterLocatorReconnects(ClusterConfigLocatorRestartDUnitTest.java:90) Caused by: org.awaitility.core.ConditionTimeoutException: Condition with org.apache.geode.management.internal.configuration.ClusterConfigLocatorRestartDUnitTest was not fulfilled within 5 minutes. at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:166) at org.awaitility.core.CallableCondition.await(CallableCondition.java:78) at org.awaitility.core.CallableCondition.await(CallableCondition.java:26) at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:939) at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:908) at org.apache.geode.management.internal.configuration.ClusterConfigLocatorRestartDUnitTest.lambda$waitForLocatorToReconnect$f182e747$2(ClusterConfigLocatorRestartDUnitTest.java:226) 8300 tests completed, 1 failed, 413 skipped
VM2 mentions that cluster membership has failed.
[vm2] [info 2021/09/25 09:36:44.487 UTC server-2 <DisconnectThread> tid=0x153] cluster membership failed due to [vm2] org.apache.geode.distributed.internal.membership.api.MemberDisconnectedException: for testing [vm2] at org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.forceDisconnect(GMSMembership.java:1787) [vm2] at org.apache.geode.distributed.internal.membership.api.MembershipManagerHelper.crashDistributedSystem(MembershipManagerHelper.java:139) [vm2] at org.apache.geode.test.junit.rules.MemberStarterRule.forceDisconnectMember(MemberStarterRule.java:568) [vm2] at org.apache.geode.test.dunit.rules.MemberVM.lambda$forceDisconnect$bb17a952$1(MemberVM.java:90) [vm2] at org.apache.geode.test.dunit.internal.IdentifiableRunnable.run(IdentifiableRunnable.java:41) [vm2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [vm2] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [vm2] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [vm2] at java.lang.reflect.Method.invoke(Method.java:498) [vm2] at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123) [vm2] at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78) [vm2] at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) [vm2] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [vm2] at java.lang.reflect.Method.invoke(Method.java:498) [vm2] at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) [vm2] at sun.rmi.transport.Transport$1.run(Transport.java:200) [vm2] at sun.rmi.transport.Transport$1.run(Transport.java:197) [vm2] at java.security.AccessController.doPrivileged(Native Method) [vm2] at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [vm2] at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) [vm2] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) [vm2] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) [vm2] at java.security.AccessController.doPrivileged(Native Method) [vm2] at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) [vm2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [vm2] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [vm2] at java.lang.Thread.run(Thread.java:748)
VM0 hits address already in use.
[vm0] [info 2021/09/25 09:36:53.762 UTC locator-0 <Location services restart thread> tid=0x60] attempt to restart location services terminated [vm0] java.net.BindException: Failed to create server socket on 0.0.0.0/0.0.0.0[43091] [vm0] at org.apache.geode.distributed.internal.tcpserver.ClusterSocketCreatorImpl.createServerSocket(ClusterSocketCreatorImpl.java:75) [vm0] at org.apache.geode.internal.net.SCClusterSocketCreator.createServerSocket(SCClusterSocketCreator.java:55) [vm0] at org.apache.geode.distributed.internal.tcpserver.ClusterSocketCreatorImpl.createServerSocket(ClusterSocketCreatorImpl.java:54) [vm0] at org.apache.geode.distributed.internal.tcpserver.TcpServer.initializeServerSocket(TcpServer.java:196) [vm0] at org.apache.geode.distributed.internal.tcpserver.TcpServer.startServerThread(TcpServer.java:183) [vm0] at org.apache.geode.distributed.internal.tcpserver.TcpServer.restarting(TcpServer.java:161) [vm0] at org.apache.geode.distributed.internal.membership.gms.locator.MembershipLocatorImpl.restarting(MembershipLocatorImpl.java:144) [vm0] at org.apache.geode.distributed.internal.InternalLocator.restartWithoutSystem(InternalLocator.java:1200) [vm0] at org.apache.geode.distributed.internal.InternalLocator.attemptReconnect(InternalLocator.java:1144) [vm0] at org.apache.geode.distributed.internal.InternalLocator.lambda$launchRestartThread$4(InternalLocator.java:1077) [vm0] at java.lang.Thread.run(Thread.java:748) [vm0] Caused by: java.net.BindException: Address already in use (Bind failed) [vm0] at java.net.PlainSocketImpl.socketBind(Native Method) [vm0] at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) [vm0] at java.net.ServerSocket.bind(ServerSocket.java:390) [vm0] at org.apache.geode.distributed.internal.tcpserver.ClusterSocketCreatorImpl.createServerSocket(ClusterSocketCreatorImpl.java:72) [vm0] ... 10 more