Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9070

Kafka Connect - Get the wrong offset value comes from Kafka Connect after increase the number of offset storage topic partition to 3

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:
    • Environment:
      debezium/connect: 0.10
      mysql: 5.6
      kafka: 2.3.0

      Description

      I'm using the Debezium project with distributed mode.

      Problem:

      I found a problem : Get the wrong offset value comes from Kafka Connect after increase the number of offset storage topic partition to 3.

      1. Cluster mode, tow node (container);
      2. Offset storage topic only have 1 partition;
      3. Create 3 mysql connector ;
      4. Change data, all connector have offset recored in partition 0.
      4. Increase offset storage topic partition number to 3.
      5. Change data, and some connector store their offset records to partition 1 or partition 2.
      6.* Restart all connect service, then all connector will read offset records from partition 0*.
      7. Then, the connector that stored offset records to partition 1 or partition 2 will get repeat data. (the offset records in partition 0 are* too old* for this connectors)

      Debug
      The Debezium developer checked their code found that the partition handling is delegated to Kafka Connect. their reply

      Then, after they reproduced the problem, they found that the problem is that the incorrect offset is returned by Kafka Connect .

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kaikai.hou kaikai.hou
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: