Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4667

KIP-154: Connect should create internal topics

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      I'm reporting this as an issue but in fact it requires more investigation (which unfortunately I'm not able to perform at this time).

      Repro steps:

      • configure Kafka for consistency, for example:
        default.replication.factor=3
        min.insync.replicas=2
        unclean.leader.election.enable=false
      • run Connect for the first time, which should create its internal topics

      I believe these topics are created with the broker's default, in particular:
      min.insync.replicas=2
      unclean.leader.election.enable=false
      but connect doesn't produce with acks=all, which in turn may cause the cluster to go in a bad state (see, e.g., https://issues.apache.org/jira/browse/KAFKA-4666).

      Solution would be to force availability mode, i.e. force:
      unclean.leader.election.enable=true
      when creating the connect topics, or viceversa detect availability vs consistency mode and turn acks=all if needed.

      I assume the same happens with other kafka-based services such as streams.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rhauch Randall Hauch
                Reporter:
                ecesena Emanuele Cesena
                Reviewer:
                Ewen Cheslack-Postava
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: