[KAFKA-13338] kafka upgrade 6.1.1 asking to change log.cleanup.policy from delete to compact

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Component/s: log
• Assignee: Unassigned
• Reporter: Fátima Galera
• Created: 01/Oct/21 16:01
• Updated: 12/Feb/24 21:55
• Resolved: 12/Feb/24 21:50

Description

Hi all,

 

During kafka upgrade from 5.3.1 to 6.1.1.1 we get below error starting kafka-connector services

 

ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
87738-org.apache.kafka.common.config.ConfigException: Topic 'connect-offsets' supplied via the 'offset.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of source connector offsets, but found the topic currently has 'cleanup.policy=delete'. Continuing would likely result in eventually losing source connector offsets and problems restarting this Connect cluster in the future. Change the 'offset.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.

 

After that error we added log.cleanup.policy=compact parameter on /etc/kafka/server.properties and kafka-connector service started but a few days later all the connectors are down with attached error.

 

Could you please let us know how we can resolved the issue?

 

Best regards and thanks

Fátima

NGA -Alight