[KAFKA-4667] KIP-154: Connect should create internal topics

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.11.0.0
• Component/s: connect
• Assignee: Randall Hauch
• Reporter: Emanuele Cesena
• Reviewer: Ewen Cheslack-Postava
• Created: 17/Jan/17 22:39
• Updated: 22/Feb/23 22:56
• Resolved: 18/May/17 23:02

Description

I'm reporting this as an issue but in fact it requires more investigation (which unfortunately I'm not able to perform at this time).

Repro steps:

• configure Kafka for consistency, for example:
  default.replication.factor=3
  min.insync.replicas=2
  unclean.leader.election.enable=false
• run Connect for the first time, which should create its internal topics

I believe these topics are created with the broker's default, in particular:
min.insync.replicas=2
unclean.leader.election.enable=false
but connect doesn't produce with acks=all, which in turn may cause the cluster to go in a bad state (see, e.g., https://issues.apache.org/jira/browse/KAFKA-4666).

Solution would be to force availability mode, i.e. force:
unclean.leader.election.enable=true
when creating the connect topics, or viceversa detect availability vs consistency mode and turn acks=all if needed.

I assume the same happens with other kafka-based services such as streams.

Issue Links

• fixes: KAFKA-4006 Kafka connect fails sometime with InvalidTopicException in distributed mode
• is duplicated by: KAFKA-5087 Kafka Connect's configuration topics should always be compacted
• links to: GitHub Pull Request #2984
• mentioned in: Page Loading...