[KAFKA-9216] Enforce connect internal topic configuration at startup

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0.0
• Fix Version/s: 2.3.2, 2.6.0, 2.4.2, 2.5.1
• Component/s: connect
• Assignee: Evelyn Bayes
• Reporter: Randall Hauch
• Reviewer: Randall Hauch
• Created: 20/Nov/19 16:49
• Updated: 11/Jun/20 05:10
• Resolved: 11/Jun/20 05:10

Description

Users sometimes configure Connect's internal topic for configurations with more than one partition. One partition is expected, however, and using more than one leads to weird behavior that is sometimes not easy to spot.

Here's one example of a log message:

"textPayload": "[2019-11-20 11:12:14,049] INFO [Worker clientId=connect-1, groupId=td-connect-server] Current config state offset 284 does not match group assignment 274. Forcing rebalance. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)\n"

Would it be possible to add a check in the KafkaConfigBackingStore and prevent the worker from starting if connect config partition count !=1 ?

Issue Links

• fixes: KAFKA-9468 config.storage.topic partition count issue is hard to debug
• is related to: KAFKA-9468 config.storage.topic partition count issue is hard to debug
• supercedes: KAFKA-5087 Kafka Connect's configuration topics should always be compacted
• links to: GitHub Pull Request #8270
• links to: GitHub Pull Request #8828