Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7745

Kafka Connect doesn't create tombstone record for tasks of deleted connector

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.1.1
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:

      Description

      Test case

      1. Create a connector:
        curl -i -XPOST -H "Accept:application/json" -H  "Content-Type:application/json" http://localhost:8083/connectors/ -d'
        {
          "name": "postgres-connector",
          "config": {
            "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
            "database.hostname": "postgresql",
            "database.port": "5432",
            "database.user": "postgres",
            "database.password": "postgres",
            "database.dbname": "postgres",
            "database.server.name": "test",
            "slot.drop_on_stop": "true"
          }
        }'
        
      2. Delete the connector:
        curl -XDELETE http://localhost:8083/connectors/postgres-connector
        
      3. Check your config topic of Kafka Connect:
        ./bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic my-connect-config --from-beginning --property print.key=true
        

      AR:
      The following records are read from the config topic after the connector is created:

      Key Value
      connector-postgres-connector {"properties":{"connector.class":"io.debezium.connector.postgresql.PostgresConnector","database.hostname":"postgresql","database.port":"5432","database.user":"postgres","database.password":"postgres","database.dbname":"postgres","database.server.name":"test","slot.drop_on_stop":"true","name":"postgres-connector"}}
      task-postgres-connector-0 {"properties":{"connector.class":"io.debezium.connector.postgresql.PostgresConnector","slot.drop_on_stop":"true","database.user":"postgres","database.dbname":"postgres","task.class":"io.debezium.connector.postgresql.PostgresConnectorTask","database.hostname":"postgresql","database.password":"postgres","name":"postgres-connector","database.server.name":"test","database.port":"5432"}}
      commit-postgres-connector {"tasks":1}

      The following records are read from the config topic after the connector is deleted:

      Key Value
      connector-postgres-connector null
      target-state-postgres-connector null

      Note that the tombstone record is missing for the task-postgres-connector-0 key.

      Additionally, the following fields of org.apache.kafka.connect.storage.KafkaConfigBackingStore class are never cleared during connector removal:

      • connectorTaskCounts
      • taskConfigs
      • deferredTaskUpdates

      and can lead to memory leaks.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jchipmunk Andrey Pustovetov
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: