Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6661

Sink connectors that explicitly 'resume' topic partitions can resume a paused task

    XMLWordPrintableJSON

    Details

      Description

      Sink connectors are allowed to use the SinkTaskContext's methods to explicitly pause and resume topic partitions. This is useful when connectors need additional time processing the records for specific topic partitions (e.g., the external system has an outage).

      However, when the sink connector has been paused via the REST API, the worker for the sink tasks pause the consumer. When the connector is polled, the poll request might timeout and return no records. Connect then calls the task's put(...) method (with no records), and this allows the task to optionally call any of the SinkTaskContext's pause or resume methods. If it calls resume, this will unexpectedly resume the paused consumer, causing the consumer to return messages and the connector to process those messages – despite the connector still being paused.

      This is reported against 1.0, but the affected code has not been changed since at least 0.9.0.0.

      A workaround is to remove rather than pause a connector. It's inconvenient, but it works.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rhauch Randall Hauch
                Reporter:
                rhauch Randall Hauch
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: