[HDDS-6122] Intermittent port conflict in TestSecureContainerServer

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Component/s: test
• Target Version/s: 1.3.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 17/Dec/21 17:53
• Updated: 08/Dec/23 11:15
• Resolved: 26/Jul/22 08:16

Description

Occasionally two datanodes of a pipeline get the same random port, causing test failure:

2021-08-25 17:10:07,267 [grpc-default-executor-0] INFO  server.RaftServer (RaftServerProxy.java:addNew(95)) - 993e5684-af6a-4cd3-a9e1-7793113d7564: addNew group-F986EA88182C:[993e5684-af6a-4cd3-a9e1-7793113d7564|rpc:0.0.0.0:41775|dataStream:|priority:0, 7bfea596-d65a-4d02-a361-749d82551ec7|rpc:0.0.0.0:35363|dataStream:|priority:0, 854bed81-cbba-43a6-9a51-b9d683b3208d|rpc:0.0.0.0:41775|dataStream:|priority:0] ...
...
2021-08-25 17:10:07,446 [main] ERROR server.GrpcService (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to start Grpc server
java.io.IOException: Failed to bind
	at org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:264)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
	at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:249)
	at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:270)
	at org.apache.ratis.server.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:72)
	at org.apache.ratis.server.impl.RaftServerProxy.lambda$start$5(RaftServerProxy.java:375)
	at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:270)
	at org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:373)
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:486)
	at org.apache.hadoop.ozone.container.server.TestSecureContainerServer.runTestClientServer(TestSecureContainerServer.java:257)
	at org.apache.hadoop.ozone.container.server.TestSecureContainerServer.runTestClientServerRatis(TestSecureContainerServer.java:230)
	at org.apache.hadoop.ozone.container.server.TestSecureContainerServer.testClientServerRatisGrpc(TestSecureContainerServer.java:212)

https://github.com/elek/ozone-build-results/blob/master/2021/08/25/9944/it-ozone/target/integration/org.apache.hadoop.ozone.container.server.TestSecureContainerServer-output.txt
https://github.com/elek/ozone-build-results/blob/master/2021/10/04/10641/it-ozone/target/integration/org.apache.hadoop.ozone.container.server.TestSecureContainerServer-output.txt

Issue Links

• relates to: HDDS-9881 Intermittent address already in use in TestSecureContainerServer