[KAFKA-3213] [CONNECT] It looks like we are not backing off properly when reconfiguring tasks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.5.0
• Component/s: connect
• Assignee: Liquan Pei
• Reporter: Gwen Shapira
• Created: 04/Feb/16 23:03
• Updated: 12/Feb/24 21:58
• Resolved: 12/Feb/24 21:58

Description

Looking at logs of attempt to reconfigure connector while leader is restarting, I see:

[2016-01-29 20:31:01,799] ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestServer)
java.net.ConnectException: Connection refused
[2016-01-29 20:31:01,802] ERROR Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error trying to forward REST request: Connection refused
[2016-01-29 20:31:01,802] ERROR Failed to reconfigure connector's tasks, retrying after backoff: (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error trying to forward REST request: Connection refused
[2016-01-29 20:31:01,803] DEBUG Sending POST with input [{"tables":"bar","table.poll.interval.ms":"1000","incrementing.column.name":"id","connection.url":"jdbc:mysql://worker1:3306/testdb?user=root","name":"test-mysql-jdbc","tasks.max":"3","task.class":"io.confluent.connect.jdbc.JdbcSourceTask","poll.interval.ms":"1000","topic.prefix":"test-","connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","mode":"incrementing"},{"tables":"foo","table.poll.interval.ms":"1000","incrementing.column.name":"id","connection.url":"jdbc:mysql://worker1:3306/testdb?user=root","name":"test-mysql-jdbc","tasks.max":"3","task.class":"io.confluent.connect.jdbc.JdbcSourceTask","poll.interval.ms":"1000","topic.prefix":"test-","connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","mode":"incrementing"}] to http://worker2:8083/connectors/test-mysql-jdbc/tasks (org.apache.kafka.connect.runtime.rest.RestServer)
[2016-01-29 20:31:01,803] ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestServer)
java.net.ConnectException: Connection refused
[2016-01-29 20:31:01,804] ERROR Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error trying to forward REST request: Connection refused

Note that it looks like we are retrying every 1ms, while I'd expect a retry every 250ms.

Issue Links

• is fixed by: KAFKA-14732 Use an exponential backoff retry mechanism while reconfiguring connector tasks