[KAFKA-15343] Fix MirrorConnectIntegrationTests causing ci build failures.

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.6.0
• Component/s: build
• Assignee: Greg Harris
• Reporter: Prasanth Kumar
• Created: 15/Aug/23 05:57
• Updated: 14/Feb/24 20:46

Description

There are several instances of tests interacting badly with gradle daemon(s) running on ports that the kafka broker previously used. After going through the debug logs we observed a few retrying kafka clients trying to connect to broker which got shutdown and the gradle worker chose the same port on which broker was running. Later in the build, the gradle daemon attempted to connect to the worker and could not, triggering a failure. Ideally gradle would not exit when connected to from an invalid client - in testing with netcat, it would often handle these without dying. However there appear to be some cases where the daemon dies completely. Both the broker code and the gradle workers bind to port 0, resulting in the OS assigning it an unused port. This does avoid conflicts, but does not ensure that long lived clients do not attempt to connect to these ports afterwards. It's possible that closing the client in between may be enough to work around this issue. Till then we will disable the test to avoid the ci blocker from testing the code changes.

MirrorConnectorsIntegrationBaseTest and extending Tests

[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+0000 [DEBUG] [TestEventLogger] 
MirrorConnectorsWithCustomForwardingAdminIntegrationTest > testReplicateSourceDefault() STANDARD_OUT
[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+0000 [DEBUG] [TestEventLogger]     [2023-07-04 11:47:46,799]
 INFO primary REST service: http://localhost:43809/connectors (org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest:224)
[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+0000 [DEBUG] [TestEventLogger]     [2023-07-04 11:47:46,799] 
INFO backup REST service: http://localhost:43323/connectors (org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest:225)
[2023-07-04T11:48:16.128Z] 2023-07-04T11:47:46.804+0000 [DEBUG] [TestEventLogger]     [2023-07-04 11:47:46,799] 
INFO primary brokers: localhost:37557 (org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest:226)
[2023-07-04T11:59:12.968Z] 2023-07-04T11:59:12.900+0000 [DEBUG] [org.gradle.internal.remote.internal.inet.TcpIncomingConnector] 
Accepted connection from /127.0.0.1:47660 to /127.0.0.1:37557.
[2023-07-04T11:59:13.233Z] 
org.gradle.internal.remote.internal.MessageIOException: Could not read message from '/127.0.0.1:47660'.
[2023-07-04T11:59:12.970Z] 2023-07-04T11:59:12.579+0000 [DEBUG] 
[org.gradle.internal.remote.internal.inet.TcpIncomingConnector] Listening on [d6bf30cb-bca2-46d9-8aeb-b9fd0497f54d port:37557, addresses:[localhost/127.0.0.1]].
[2023-07-04T11:59:46.519Z] 2023-07-04T11:59:13.014+0000 [ERROR] 
[system.err] org.gradle.internal.remote.internal.ConnectException: Could not connect to server [d6bf30cb-bca2-46d9-8aeb-b9fd0497f54d port:37557, addresses:[/127.0.0.1]]. Tried addresses: [/127.0.0.1]. 

Issue Links

• is duplicated by: KAFKA-15333 Flaky build failure throwing Connect Exception: Could not connect to server....
• is related to: KAFKA-15845 Add Junit5 test extension which detects leaked Kafka clients and servers